{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/godiva-generating-open-domain-videos-from","title":"GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions","arxiv_id":"2104.14806","date":"2021-04-30","proceeding":null,"authors":["Chenfei Wu","Lun Huang","Qianxi Zhang","Binyang Li","Lei Ji","Fan Yang","Guillermo Sapiro","Nan Duan"],"abstract":"Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.","url_abs":"https://arxiv.org/abs/2104.14806v1","url_pdf":"https://arxiv.org/pdf/2104.14806v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"godiva-generating-open-domain-videos-from","repo_url":"https://github.com/mehdidc/DALLE_clip_score","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-video-generation-on-msr-vtt","task":"Text-to-Video Generation","dataset":"MSR-VTT","model":"GODIVA","rank_in_archive_order":18,"of":18,"metrics":{"CLIPSIM":"0.2402"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.14806","atlas_url":"https://app.syntology.ai/?focus=2104.14806","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}