{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-language-models-as-consistent-story","title":"StoryGPT-V: Large Language Models as Consistent Story Visualizers","arxiv_id":"2312.02252","date":"2023-12-04","proceeding":"CVPR 2025 1","authors":["Xiaoqian Shen","Mohamed Elhoseiny"],"abstract":"Recent generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts. Nevertheless, a significant challenge remains in applying these models for the more intricate task of story visualization. Since it requires resolving pronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution, and ensuring consistent characters and background synthesis across frames. Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references and process extensive sequences. Therefore, we introduce \\textbf{StoryGPT-V}, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters grounded on given story descriptions. First, we train a character-aware LDM, which takes character-augmented semantic embedding as input and includes the supervision of the cross-attention map using character segmentation masks, aiming to enhance character generation accuracy and faithfulness. In the second stage, we enable an alignment between the output of LLM and the character-augmented embedding residing in the input space of the first-stage model. This harnesses the reasoning ability of LLM to address ambiguous references and the comprehension capability to memorize the context. We conduct comprehensive experiments on two visual story visualization benchmarks. Our model reports superior quantitative results and consistently generates accurate characters of remarkable quality with low memory consumption. Our code will be made publicly available.","url_abs":"https://arxiv.org/abs/2312.02252v2","url_pdf":"https://arxiv.org/pdf/2312.02252v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-language-models-as-consistent-story","repo_url":"https://github.com/xiaoqian-shen/StoryGPT-V","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"story-visualization","task_name":"Story Visualization"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.02252","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02252"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiaoqian-shen/StoryGPT-V","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":15,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a3b4f58d84032797","entry":"collate_fn","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/data.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3b4f58d84032797"}},{"code_sha256_prefix":"83ada93fd84968cc","entry":"contrastive_loss","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"gill/losses.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/gill/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83ada93fd84968cc"}},{"code_sha256_prefix":"8d1084e28d0e8fb1","entry":"find_start_match","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/data.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d1084e28d0e8fb1"}},{"code_sha256_prefix":"697ac9abccfd1b93","entry":"get_reg_loss_layer","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"697ac9abccfd1b93"}},{"code_sha256_prefix":"addaaf66c3ab7352","entry":"get_test_transforms","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/transforms.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"addaaf66c3ab7352"}},{"code_sha256_prefix":"c036796b62b5952d","entry":"get_train_transforms","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/transforms.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c036796b62b5952d"}},{"code_sha256_prefix":"15ed5aec7f6bd270","entry":"get_train_transforms_with_segmap","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/transforms.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15ed5aec7f6bd270"}},{"code_sha256_prefix":"817e9d69c2cd23d3","entry":"prepare_image_token_idx","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/data.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"817e9d69c2cd23d3"}},{"code_sha256_prefix":"d3c9a29ac1be57e9","entry":"stable_diffusion_call_with_references_delayed_conditioning","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3c9a29ac1be57e9"}},{"code_sha256_prefix":"eefbde796616ac35","entry":"calculate_activation_statistics","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"eval/fid_score.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/eval/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eefbde796616ac35"}},{"code_sha256_prefix":"b71963b3facc5fb5","entry":"calculate_frechet_distance","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"eval/fid_score.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/eval/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b71963b3facc5fb5"}},{"code_sha256_prefix":"2fa8e59bc1db947b","entry":"fuse_object_embeddings","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fa8e59bc1db947b"}},{"code_sha256_prefix":"7f4dc018c6466fcc","entry":"l1_loss","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"gill/losses.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/gill/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f4dc018c6466fcc"}},{"code_sha256_prefix":"f04280cd71a46d32","entry":"l2_loss","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"gill/losses.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/gill/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f04280cd71a46d32"}},{"code_sha256_prefix":"0a5a54048f832e12","entry":"unet_store_cross_attention_scores","repo":"xiaoqian-shen/StoryGPT-V","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/xiaoqian-shen/StoryGPT-V/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a5a54048f832e12"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}