{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/seed-story-multimodal-long-story-generation","title":"SEED-Story: Multimodal Long Story Generation with Large Language Model","arxiv_id":"2407.08683","date":"2024-07-11","proceeding":null,"authors":["Shuai Yang","Yuying Ge","Yang Li","Yukang Chen","Yixiao Ge","Ying Shan","Yingcong Chen"],"abstract":"With the remarkable advancements in image generation and open-form text generation, the creation of interleaved image-text content has become an increasingly intriguing field. Multimodal story generation, characterized by producing narrative texts and vivid images in an interleaved manner, has emerged as a valuable and practical task with broad applications. However, this task poses significant challenges, as it necessitates the comprehension of the complex interplay between texts and images, and the ability to generate long sequences of coherent, contextually relevant texts and visuals. In this work, we propose SEED-Story, a novel method that leverages a Multimodal Large Language Model (MLLM) to generate extended multimodal stories. Our model, built upon the powerful comprehension capability of MLLM, predicts text tokens as well as visual tokens, which are subsequently processed with an adapted visual de-tokenizer to produce images with consistent characters and styles. We further propose multimodal attention sink mechanism to enable the generation of stories with up to 25 sequences (only 10 for training) in a highly efficient autoregressive manner. Additionally, we present a large-scale and high-resolution dataset named StoryStream for training our model and quantitatively evaluating the task of multimodal story generation in various aspects.","url_abs":"https://arxiv.org/abs/2407.08683v2","url_pdf":"https://arxiv.org/pdf/2407.08683v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seed-story-multimodal-long-story-generation","repo_url":"https://github.com/tencentarc/seed-story","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"story-generation","task_name":"Story Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.08683","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08683"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencentarc/seed-story","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_fixture":3,"ran":8,"ran_draft_wrong":3,"unverified":3},"by_repo_kind":{"official":{"samples":17,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models_clm/modeling_llama_4_35.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models_clm/modeling_llama_4_35.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"b23ef17a95f4057a","entry":"add_subtitle","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/inference/gen_george.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/inference/gen_george.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b23ef17a95f4057a"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models_clm/modeling_llama_4_35.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models_clm/modeling_llama_4_35.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models_clm/modeling_llama_xformer.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models_clm/modeling_llama_xformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"66a2fe5698649c4c","entry":"cosine_loss","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models/discrete_models.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models/discrete_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"66a2fe5698649c4c"}},{"code_sha256_prefix":"2c162a9ad508c175","entry":"create_json_structure","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2c162a9ad508c175"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story_v2.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story_v2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"d75e2fd34b44dd96","entry":"extract_info_from_output","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d75e2fd34b44dd96"}},{"code_sha256_prefix":"1c2a0387fb195e14","entry":"generate_story","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1c2a0387fb195e14"}},{"code_sha256_prefix":"77e8a3ac46f3afec","entry":"get_2d_sincos_pos_embed","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models/qwen_visual.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models/qwen_visual.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77e8a3ac46f3afec"}},{"code_sha256_prefix":"50c4398a8645438a","entry":"get_abs_pos","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models/qwen_visual.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models/qwen_visual.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50c4398a8645438a"}},{"code_sha256_prefix":"a49b0eb559884898","entry":"read_json_and_extract_content","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/eval/gpt_comparative_eval.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/eval/gpt_comparative_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a49b0eb559884898"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models_clm/modeling_llama_4_35.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models_clm/modeling_llama_4_35.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"e1fb527ab9de5e39","entry":"split_entries","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/chunk_data.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/chunk_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e1fb527ab9de5e39"}},{"code_sha256_prefix":"d3a1dc8794e15989","entry":"api_call","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story_v2.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d3a1dc8794e15989"}},{"code_sha256_prefix":"4b99bf78b5f9552e","entry":"construct_dataset","repo":"tencentarc/seed-story","repo_kind":"official","path":"StoryStream/build_story_v2.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/StoryStream/build_story_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4b99bf78b5f9552e"}},{"code_sha256_prefix":"2884deadbb637c3e","entry":"get_2d_sincos_pos_embed_from_grid","repo":"tencentarc/seed-story","repo_kind":"official","path":"src/models/qwen_visual.py","file_url":"https://github.com/tencentarc/seed-story/blob/HEAD/src/models/qwen_visual.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2884deadbb637c3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}