{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/boosting-consistency-in-story-visualization","title":"Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models","arxiv_id":"2407.02482","date":"2024-07-02","proceeding":null,"authors":["Fei Shen","Hu Ye","Sibo Liu","Jun Zhang","Cong Wang","Xiao Han","Wei Yang"],"abstract":"Recent research showcases the considerable potential of conditional diffusion models for generating consistent stories. However, current methods, which predominantly generate stories in an autoregressive and excessively caption-dependent manner, often underrate the contextual consistency and relevance of frames during sequential generation. To address this, we propose a novel Rich-contextual Conditional Diffusion Models (RCDMs), a two-stage approach designed to enhance story generation's semantic consistency and temporal consistency. Specifically, in the first stage, the frame-prior transformer diffusion model is presented to predict the frame semantic embedding of the unknown clip by aligning the semantic correlations between the captions and frames of the known clip. The second stage establishes a robust model with rich contextual conditions, including reference images of the known clip, the predicted frame semantic embedding of the unknown clip, and text embeddings of all captions. By jointly injecting these rich contextual conditions at the image and feature levels, RCDMs can generate semantic and temporal consistency stories. Moreover, RCDMs can generate consistent stories with a single forward inference compared to autoregressive models. Our qualitative and quantitative results demonstrate that our proposed RCDMs outperform in challenging scenarios. The code and model will be available at https://github.com/muzishen/RCDMs.","url_abs":"https://arxiv.org/abs/2407.02482v2","url_pdf":"https://arxiv.org/pdf/2407.02482v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"boosting-consistency-in-story-visualization","repo_url":"https://github.com/muzishen/rcdms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"story-visualization","task_name":"Story Visualization"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.02482","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.02482"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/muzishen/rcdms","reach":null}],"summary":{"ran_honours":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"4b208faf97544161","entry":"count_model_params","repo":"muzishen/rcdms","repo_kind":"official","path":"train_stage1.py","file_url":"https://github.com/muzishen/rcdms/blob/HEAD/train_stage1.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b208faf97544161"}},{"code_sha256_prefix":"ff3fab764a3fa17d","entry":"load_training_checkpoint","repo":"muzishen/rcdms","repo_kind":"official","path":"train_stage2.py","file_url":"https://github.com/muzishen/rcdms/blob/HEAD/train_stage2.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff3fab764a3fa17d"}},{"code_sha256_prefix":"c5e2ee446146ae08","entry":"mask2list_label","repo":"muzishen/rcdms","repo_kind":"official","path":"train_stage2.py","file_url":"https://github.com/muzishen/rcdms/blob/HEAD/train_stage2.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5e2ee446146ae08"}},{"code_sha256_prefix":"4d5366154562b512","entry":"load_training_checkpoint","repo":"muzishen/rcdms","repo_kind":"official","path":"train_stage1.py","file_url":"https://github.com/muzishen/rcdms/blob/HEAD/train_stage1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d5366154562b512"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}