{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/storybench-a-multifaceted-benchmark-for-1","title":"StoryBench: A Multifaceted Benchmark for Continuous Story Visualization","arxiv_id":"2308.11606","date":"2023-08-22","proceeding":"NeurIPS 2023 11","authors":["Emanuele Bugliarello","Hernan Moraldo","Ruben Villegas","Mohammad Babaeizadeh","Mohammad Taghi Saffar","Han Zhang","Dumitru Erhan","Vittorio Ferrari","Pieter-Jan Kindermans","Paul Voigtlaender"],"abstract":"Generating video stories from text prompts is a complex task. In addition to having high visual quality, videos need to realistically adhere to a sequence of text prompts whilst being consistent throughout the frames. Creating a benchmark for video generation requires data annotated over time, which contrasts with the single caption used often in video datasets. To fill this gap, we collect comprehensive human annotations on three existing datasets, and introduce StoryBench: a new, challenging multi-task benchmark to reliably evaluate forthcoming text-to-video models. Our benchmark includes three video generation tasks of increasing difficulty: action execution, where the next action must be generated starting from a conditioning video; story continuation, where a sequence of actions must be executed starting from a conditioning video; and story generation, where a video must be generated from only text prompts. We evaluate small yet strong text-to-video baselines, and show the benefits of training on story-like data algorithmically generated from existing video captions. Finally, we establish guidelines for human evaluation of video stories, and reaffirm the need of better automatic metrics for video generation. StoryBench aims at encouraging future research efforts in this exciting new area.","url_abs":"https://arxiv.org/abs/2308.11606v2","url_pdf":"https://arxiv.org/pdf/2308.11606v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"storybench-a-multifaceted-benchmark-for-1","repo_url":"https://github.com/google/storybench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"story-continuation","task_name":"Story Continuation"},{"task_slug":"story-generation","task_name":"Story Generation"},{"task_slug":"story-visualization","task_name":"Story Visualization"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.11606","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.11606"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/google/storybench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran":11,"unverified":2},"by_repo_kind":{"official":{"samples":16,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/simple_tokenizer.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"ae2774392b2c159e","entry":"check_gpu","repo":"google/storybench","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae2774392b2c159e"}},{"code_sha256_prefix":"bc39b8ed5fefa7eb","entry":"crop_clip","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/video_transform.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/video_transform.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc39b8ed5fefa7eb"}},{"code_sha256_prefix":"6c1abcbfdccc77d3","entry":"get_frechet_dist","repo":"google/storybench","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c1abcbfdccc77d3"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/simple_tokenizer.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"f1a347e31ccb9916","entry":"get_resize_function","repo":"google/storybench","repo_kind":"official","path":"metrics/models/dover/datasets.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/dover/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1a347e31ccb9916"}},{"code_sha256_prefix":"2b53e457d5632b4e","entry":"get_resize_sizes","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/video_transform.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/video_transform.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b53e457d5632b4e"}},{"code_sha256_prefix":"aba6f7587829bce5","entry":"get_resized_video","repo":"google/storybench","repo_kind":"official","path":"metrics/models/dover/datasets.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/dover/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aba6f7587829bce5"}},{"code_sha256_prefix":"549ed6c630ddf0dc","entry":"get_spatial_fragments","repo":"google/storybench","repo_kind":"official","path":"metrics/models/dover/datasets.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/dover/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"549ed6c630ddf0dc"}},{"code_sha256_prefix":"06974c0cf54c1fbd","entry":"load_model","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/internvideo.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/internvideo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06974c0cf54c1fbd"}},{"code_sha256_prefix":"f492906c56e978ff","entry":"load_npz","repo":"google/storybench","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f492906c56e978ff"}},{"code_sha256_prefix":"0934a13d9302e0be","entry":"resize_clip","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/video_transform.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/video_transform.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0934a13d9302e0be"}},{"code_sha256_prefix":"b482b6a0ac227ae9","entry":"score","repo":"google/storybench","repo_kind":"official","path":"metrics/vtm_clip.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/vtm_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b482b6a0ac227ae9"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/simple_tokenizer.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"fd72b75b4e4ab8a6","entry":"compute_scores","repo":"google/storybench","repo_kind":"official","path":"metrics/pqa_dover.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/pqa_dover.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd72b75b4e4ab8a6"}},{"code_sha256_prefix":"962c20d80a29a8c4","entry":"preprocess_video","repo":"google/storybench","repo_kind":"official","path":"metrics/models/InternVideo/internvideo.py","file_url":"https://github.com/google/storybench/blob/HEAD/metrics/models/InternVideo/internvideo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"962c20d80a29a8c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}