{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/storydall-e-adapting-pretrained-text-to-image","title":"StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation","arxiv_id":"2209.06192","date":"2022-09-13","proceeding":null,"authors":["Adyasha Maharana","Darryl Hannan","Mohit Bansal"],"abstract":"Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives. Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters. Then, we enhance or 'retro-fit' the pretrained text-to-image synthesis models with task-specific modules for (a) sequential image generation and (b) copying relevant elements from an initial frame. Then, we explore full-model finetuning, as well as prompt-based tuning for parameter-efficient adaptation, of the pre-trained model. We evaluate our approach StoryDALL-E on two existing datasets, PororoSV and FlintstonesSV, and introduce a new dataset DiDeMoSV collected from a video-captioning dataset. We also develop a model StoryGANc based on Generative Adversarial Networks (GAN) for story continuation, and compare it with the StoryDALL-E model to demonstrate the advantages of our approach. We show that our retro-fitting approach outperforms GAN-based models for story continuation and facilitates copying of visual elements from the source image, thereby improving continuity in the generated visual story. Finally, our analysis suggests that pretrained transformers struggle to comprehend narratives containing several characters. Overall, our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation.","url_abs":"https://arxiv.org/abs/2209.06192v1","url_pdf":"https://arxiv.org/pdf/2209.06192v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"storydall-e-adapting-pretrained-text-to-image","repo_url":"https://github.com/adymaharana/storydalle","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"story-continuation","task_name":"Story Continuation"},{"task_slug":"story-visualization","task_name":"Story Visualization"},{"task_slug":"video-captioning","task_name":"Video Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/story-continuation-on-flintstonessv","task":"Story Continuation","dataset":"FlintstonesSV","model":"StoryDALL-E","rank_in_archive_order":3,"of":6,"metrics":{"Char-F1":"74.28","F-Acc":"52.35","FID":"28.37"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-flintstonessv","task":"Story Continuation","dataset":"FlintstonesSV","model":"StoryDALL-E (Story Embeddings)","rank_in_archive_order":4,"of":6,"metrics":{"Char-F1":"72.18","F-Acc":"53.28","FID":"29.21"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-flintstonessv","task":"Story Continuation","dataset":"FlintstonesSV","model":"StoryDALL-E (Cross-Attention)","rank_in_archive_order":5,"of":6,"metrics":{"Char-F1":"73.94","F-Acc":"52.72","FID":"35.04"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-flintstonessv","task":"Story Continuation","dataset":"FlintstonesSV","model":"StoryDALL-E (Story Embeddings + Cross-Attention)","rank_in_archive_order":6,"of":6,"metrics":{"Char-F1":"72.44","F-Acc":"51.32","FID":"36.28"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-pororosv","task":"Story Continuation","dataset":"PororoSV","model":"StoryDALL-E","rank_in_archive_order":3,"of":6,"metrics":{"Char-F1":"40.28","F-Acc":"20.94","FID":"21.64"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-pororosv","task":"Story Continuation","dataset":"PororoSV","model":"StoryDALL-E (Cross-Attention)","rank_in_archive_order":4,"of":6,"metrics":{"Char-F1":"40.25","F-Acc":"18.16","FID":"23.27"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-pororosv","task":"Story Continuation","dataset":"PororoSV","model":"StoryDALL-E (Story Embeddings)","rank_in_archive_order":5,"of":6,"metrics":{"Char-F1":"39.32","F-Acc":"34.65","FID":"30.45"},"uses_additional_data":false},{"leaderboard":"/sota/story-continuation-on-pororosv","task":"Story Continuation","dataset":"PororoSV","model":"StoryDALL-E (Story Embeddings + Cross-Attention)","rank_in_archive_order":6,"of":6,"metrics":{"Char-F1":"35.29","F-Acc":"16.73","FID":"31.68"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.06192","atlas_url":"https://app.syntology.ai/?focus=2209.06192","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.06192"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adymaharana/storydalle","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5f34dd7c6d84070b","entry":"Block","repo":"adymaharana/storydalle","repo_kind":"official","path":"story-dalle/dalle/models/stage2/transformer.py","file_url":"https://github.com/adymaharana/storydalle/blob/HEAD/story-dalle/dalle/models/stage2/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f34dd7c6d84070b"}},{"code_sha256_prefix":"c354cf95a9627777","entry":"MultiHeadSelfAttention","repo":"adymaharana/storydalle","repo_kind":"official","path":"story-dalle/dalle/models/stage2/transformer.py","file_url":"https://github.com/adymaharana/storydalle/blob/HEAD/story-dalle/dalle/models/stage2/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c354cf95a9627777"}},{"code_sha256_prefix":"b446857df78ea96c","entry":"Transformer1d","repo":"adymaharana/storydalle","repo_kind":"official","path":"story-dalle/dalle/models/stage2/transformer.py","file_url":"https://github.com/adymaharana/storydalle/blob/HEAD/story-dalle/dalle/models/stage2/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b446857df78ea96c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}