{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/intelligent-grimm-open-ended-visual","title":"Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models","arxiv_id":"2306.00973","date":"2023-06-01","proceeding":null,"authors":["Chang Liu","HaoNing Wu","Yujie Zhong","Xiaoyun Zhang","Yanfeng Wang","Weidi Xie"],"abstract":"Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently. In this work, we focus on a novel, yet challenging task of generating a coherent image sequence based on a given storyline, denoted as open-ended visual storytelling. We make the following three contributions: (i) to fulfill the task of visual storytelling, we propose a learning-based auto-regressive image generation model, termed as StoryGen, with a novel vision-language context module, that enables to generate the current frame by conditioning on the corresponding text prompt and preceding image-caption pairs; (ii) to address the data shortage of visual storytelling, we collect paired image-text sequences by sourcing from online videos and open-source E-books, establishing processing pipeline for constructing a large-scale dataset with diverse characters, storylines, and artistic styles, named StorySalon; (iii) Quantitative experiments and human evaluations have validated the superiority of our StoryGen, where we show StoryGen can generalize to unseen characters without any optimization, and generate image sequences with coherent content and consistent character. Code, dataset, and models are available at https://haoningwu3639.github.io/StoryGen_Webpage/","url_abs":"https://arxiv.org/abs/2306.00973v3","url_pdf":"https://arxiv.org/pdf/2306.00973v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"intelligent-grimm-open-ended-visual","repo_url":"https://github.com/haoningwu3639/StoryGen","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"story-visualization","task_name":"Story Visualization"},{"task_slug":"style-transfer","task_name":"Style Transfer"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"visual-storytelling","task_name":"Visual Storytelling"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.00973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoningwu3639/StoryGen","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"445d524b63b8a33b","entry":"calc_probs","repo":"haoningwu3639/StoryGen","repo_kind":"official","path":"inference_COCO_val.py","file_url":"https://github.com/haoningwu3639/StoryGen/blob/HEAD/inference_COCO_val.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"445d524b63b8a33b"}},{"code_sha256_prefix":"d871492da2e5fdf7","entry":"cosine_similarity","repo":"haoningwu3639/StoryGen","repo_kind":"official","path":"data_process/align.py","file_url":"https://github.com/haoningwu3639/StoryGen/blob/HEAD/data_process/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d871492da2e5fdf7"}},{"code_sha256_prefix":"02ade4b6ebe103d4","entry":"time_dist","repo":"haoningwu3639/StoryGen","repo_kind":"official","path":"data_process/align.py","file_url":"https://github.com/haoningwu3639/StoryGen/blob/HEAD/data_process/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02ade4b6ebe103d4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}