{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/imagine-this-scripts-to-compositions-to","title":"Imagine This! Scripts to Compositions to Videos","arxiv_id":"1804.03608","date":"2018-04-10","proceeding":"ECCV 2018 9","authors":["Tanmay Gupta","Dustin Schwenk","Ali Farhadi","Derek Hoiem","Aniruddha Kembhavi"],"abstract":"Imagining a scene described in natural language with realistic layout and\nappearance of entities is the ultimate test of spatial, visual, and semantic\nworld knowledge. Towards this goal, we present the Composition, Retrieval, and\nFusion Network (CRAFT), a model capable of learning this knowledge from\nvideo-caption data and applying it while generating videos from novel captions.\nCRAFT explicitly predicts a temporal-layout of mentioned entities (characters\nand objects), retrieves spatio-temporal entity segments from a video database\nand fuses them to generate scene videos. Our contributions include sequential\ntraining of components of CRAFT while jointly modeling layout and appearances,\nand losses that encourage learning compositional representations for retrieval.\nWe evaluate CRAFT on semantic fidelity to caption, composition consistency, and\nvisual quality. CRAFT outperforms direct pixel generation approaches and\ngeneralizes well to unseen captions and to unseen video databases with no text\nannotations. We demonstrate CRAFT on FLINTSTONES, a new richly annotated\nvideo-caption dataset with over 25000 videos. For a glimpse of videos generated\nby CRAFT, see https://youtu.be/688Vv86n0z8.","url_abs":"http://arxiv.org/abs/1804.03608v1","url_pdf":"http://arxiv.org/pdf/1804.03608v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"imagine-this-scripts-to-compositions-to","repo_url":"https://github.com/Leiree/Paginas-importantes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"imagine-this-scripts-to-compositions-to","repo_url":"https://github.com/ghost1412/CRAFT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"imagine-this-scripts-to-compositions-to","repo_url":"https://github.com/ishandutta2007/Craft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"imagine-this-scripts-to-compositions-to","repo_url":"https://github.com/ubc-vision/make-a-story","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"imagine-this-scripts-to-compositions-to","repo_url":"https://github.com/xiaoqian-shen/StoryGPT-V","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.03608","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}