{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vript-a-video-is-worth-thousands-of-words","title":"Vript: A Video Is Worth Thousands of Words","arxiv_id":"2406.06040","date":"2024-06-10","proceeding":null,"authors":["Dongjie Yang","Suyuan Huang","Chengqiang Lu","Xiaodong Han","Haoxin Zhang","Yan Gao","Yao Hu","Hai Zhao"],"abstract":"Advancements in multimodal learning, particularly in video understanding and generation, require high-quality video-text datasets for improved model performance. Vript addresses this issue with a meticulously annotated corpus of 12K high-resolution videos, offering detailed, dense, and script-like captions for over 420K clips. Each clip has a caption of ~145 words, which is over 10x longer than most video-text datasets. Unlike captions only documenting static content in previous datasets, we enhance video captioning to video scripting by documenting not just the content, but also the camera operations, which include the shot types (medium shot, close-up, etc) and camera movements (panning, tilting, etc). By utilizing the Vript, we explore three training paradigms of aligning more text with the video modality rather than clip-caption pairs. This results in Vriptor, a top-performing video captioning model among open-source models, comparable to GPT-4V in performance. Vriptor is also a powerful model capable of end-to-end generation of dense and detailed captions for long videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three video understanding tasks that are more challenging than existing benchmarks: Vript-HAL is the first benchmark evaluating action and object hallucinations in video LLMs, Vript-RR combines reasoning with retrieval resolving question ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the temporal understanding of events in long videos rather than actions in short videos in previous works. All code, models, and datasets are available in https://github.com/mutonix/Vript. PS: We have included more video-text datasets (Vript_CN & Vript_Multilingual) in the Vript series.","url_abs":"https://arxiv.org/abs/2406.06040v2","url_pdf":"https://arxiv.org/pdf/2406.06040v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vript-a-video-is-worth-thousands-of-words","repo_url":"https://github.com/mutonix/vript","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06040","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06040"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mutonix/Vript","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mutonix/vript","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2,"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"748d5a2586ca3178","entry":"get_prompt","repo":"mutonix/Vript","repo_kind":"official","path":"vript-hard/models/videochat2/videochat2_vriptCAP.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vript-hard/models/videochat2/videochat2_vriptCAP.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"748d5a2586ca3178"}},{"code_sha256_prefix":"db17c3cf4c7f44a9","entry":"get_prompt2","repo":"mutonix/Vript","repo_kind":"official","path":"vript-hard/models/videochat2/videochat2_vriptCAP.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vript-hard/models/videochat2/videochat2_vriptCAP.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db17c3cf4c7f44a9"}},{"code_sha256_prefix":"6cf5d1c74946c834","entry":"merge_voiceovers","repo":"mutonix/Vript","repo_kind":"official","path":"vriptor/build_vript_training_data/build_training_vript_stage1_concat.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vriptor/build_vript_training_data/build_training_vript_stage1_concat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6cf5d1c74946c834"}},{"code_sha256_prefix":"2cd2b223ddbe3624","entry":"post_process_gpt_response","repo":"mutonix/Vript","repo_kind":"official","path":"vript-hard/verification_pipeline/verify_vript-rr_open_answer.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vript-hard/verification_pipeline/verify_vript-rr_open_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2cd2b223ddbe3624"}},{"code_sha256_prefix":"fa1225dfac92bc0d","entry":"get_peft_state_maybe_zero_3","repo":"mutonix/Vript","repo_kind":"official","path":"vriptor/train_hf.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vriptor/train_hf.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1225dfac92bc0d"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"mutonix/Vript","repo_kind":"official","path":"vriptor/train_hf.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vriptor/train_hf.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"mutonix/Vript","repo_kind":"official","path":"vriptor/train_hf.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vriptor/train_hf.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"980b8c8c57db1559","entry":"num_tokens_consumed_from_request","repo":"mutonix/Vript","repo_kind":"official","path":"vript-hard/verification_pipeline/verify_vript-rr_open_answer.py","file_url":"https://github.com/mutonix/Vript/blob/HEAD/vript-hard/verification_pipeline/verify_vript-rr_open_answer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"980b8c8c57db1559"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}