{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/videollama-2-advancing-spatial-temporal","title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","arxiv_id":"2406.07476","date":"2024-06-11","proceeding":null,"authors":["Zesen Cheng","Sicong Leng","Hang Zhang","Yifei Xin","Xin Li","Guanzheng Chen","Yongxin Zhu","Wenqi Zhang","Ziyang Luo","Deli Zhao","Lidong Bing"],"abstract":"In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. Building upon its predecessor, VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC) connector, which effectively captures the intricate spatial and temporal dynamics of video data. Additionally, we integrate an Audio Branch into the model through joint training, thereby enriching the multimodal understanding capabilities of the model by seamlessly incorporating audio cues. Comprehensive evaluations on multiple-choice video question answering (MC-VQA), open-ended video question answering (OE-VQA), and video captioning (VC) tasks demonstrate that VideoLLaMA 2 consistently achieves competitive results among open-source models and even gets close to some proprietary models on several benchmarks. Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models. These advancements underline VideoLLaMA 2's superior performance in multimodal comprehension, setting a new standard for intelligent video analysis systems. All models are public to facilitate further research.","url_abs":"https://arxiv.org/abs/2406.07476v3","url_pdf":"https://arxiv.org/pdf/2406.07476v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"videollama-2-advancing-spatial-temporal","repo_url":"https://github.com/damo-nlp-sg/videollama2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"videollama-2-advancing-spatial-temporal","repo_url":"https://github.com/damo-nlp-sg/inf-clip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"videollama-2-advancing-spatial-temporal","repo_url":"https://github.com/damo-nlp-sg/videollama3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"temporal-relation-extraction","task_name":"Temporal Relation Extraction"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/temporal-relation-extraction-on-vinoground","task":"Temporal Relation Extraction","dataset":"Vinoground","model":"VideoLLaMA2-72B","rank_in_archive_order":8,"of":24,"metrics":{"Group Score":"8.4","Text Score":"36.2","Video Score":"21.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-mvbench","task":"Video Question Answering","dataset":"MVBench","model":"VideoLLaMA2 (72B)","rank_in_archive_order":6,"of":22,"metrics":{"Avg.":"62.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"VideoLLaMA2.1(7B)","rank_in_archive_order":23,"of":47,"metrics":{"Accuracy":"75.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-perception-test","task":"Video Question Answering","dataset":"Perception Test","model":"VideoLLaMA2 (72B)","rank_in_archive_order":4,"of":6,"metrics":{"Accuracy (Top-1)":"57.5"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-tvbench","task":"Video Question Answering","dataset":"TVBench","model":"VideoLLaMA2 72B","rank_in_archive_order":14,"of":28,"metrics":{"Average Accuracy":"48.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-tvbench","task":"Video Question Answering","dataset":"TVBench","model":"VideoLLaMA2 7B","rank_in_archive_order":19,"of":28,"metrics":{"Average Accuracy":"42.9"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-tvbench","task":"Video Question Answering","dataset":"TVBench","model":"VideoLLaMA2.1","rank_in_archive_order":22,"of":28,"metrics":{"Average Accuracy":"42.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"VideoLLaMA2 (72B)","rank_in_archive_order":6,"of":29,"metrics":{"Accuracy":"63.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-vnbench","task":"Zero-Shot Video Question Answer","dataset":"VNBench","model":"VideoLLaMA2","rank_in_archive_order":8,"of":9,"metrics":{"Accuracy":"4.5"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-video-mme-1","task":"Zero-Shot Video Question Answer","dataset":"Video-MME","model":"VideoLLaMA2 (72B)","rank_in_archive_order":9,"of":11,"metrics":{"Accuracy (%)":"63.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-video-mme","task":"Zero-Shot Video Question Answer","dataset":"Video-MME (w/o subs)","model":"VideoLLaMA2 (72B)","rank_in_archive_order":8,"of":9,"metrics":{"Accuracy (%)":"60.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.07476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-nlp-sg/videollama2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-nlp-sg/videollama3","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-nlp-sg/inf-clip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran_violates":1,"ran_fixture":1,"ran":1,"unverified":10},"by_repo_kind":{"listed":{"samples":17,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/tokenizer.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/clip_arch.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/clip_arch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"4bdd36eab04c3f62","entry":"get_1d_sincos_pos_embed_from_grid","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/pos_embed.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdd36eab04c3f62"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/clip_arch.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/clip_arch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/clip_arch.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/clip_arch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/tokenizer.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"0b94bc3f6eb369a4","entry":"load_state_dict","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/factory.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b94bc3f6eb369a4"}},{"code_sha256_prefix":"86e38328c37d09a8","entry":"all_reduce","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/loss.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86e38328c37d09a8"}},{"code_sha256_prefix":"9ad02daca617f28b","entry":"gather_features","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/loss.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ad02daca617f28b"}},{"code_sha256_prefix":"9417ae492629cf06","entry":"get_2d_sincos_pos_embed","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/pos_embed.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9417ae492629cf06"}},{"code_sha256_prefix":"4fd80de79832745d","entry":"get_2d_sincos_pos_embed_from_grid","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/pos_embed.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd80de79832745d"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/factory.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"cd2f59a566701fe4","entry":"get_tokenizer","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/factory.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd2f59a566701fe4"}},{"code_sha256_prefix":"e332856e3c2fc814","entry":"neighbour_exchange","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/loss.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e332856e3c2fc814"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/coca_arch.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/coca_arch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/hf_model.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}},{"code_sha256_prefix":"62c3f3fab07e9ee3","entry":"whitespace_clean","repo":"damo-nlp-sg/inf-clip","repo_kind":"listed","path":"inf_clip/models/tokenizer.py","file_url":"https://github.com/damo-nlp-sg/inf-clip/blob/HEAD/inf_clip/models/tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62c3f3fab07e9ee3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}