{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/moviechat-question-aware-sparse-memory-for","title":"MovieChat+: Question-aware Sparse Memory for Long Video Question Answering","arxiv_id":"2404.17176","date":"2024-04-26","proceeding":null,"authors":["Enxin Song","Wenhao Chai","Tian Ye","Jenq-Neng Hwang","Xi Li","Gaoang Wang"],"abstract":"Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following https://github.com/rese1f/MovieChat.","url_abs":"https://arxiv.org/abs/2404.17176v1","url_pdf":"https://arxiv.org/pdf/2404.17176v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"moviechat-question-aware-sparse-memory-for","repo_url":"https://github.com/rese1f/MovieChat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"2k","task_name":"2k"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-next-qa-open-ended","task":"Question Answering","dataset":"NExT-QA (Open-ended VideoQA)","model":"MovieChat+","rank_in_archive_order":4,"of":6,"metrics":{"Accuracy":"54.8","Confidence Score":"3.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.17176","atlas_url":"https://app.syntology.ai/?focus=2404.17176","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.17176"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rese1f/MovieChat","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":2,"ran":5,"ran_violates":1,"ran_honours":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b4dff79d5e6102c","entry":"apply_rotary_pos_emb","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/modeling_llama.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9b4dff79d5e6102c"}},{"code_sha256_prefix":"54a7a818ecf43861","entry":"bipartite_soft_matching","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/eva_vit_with_tome.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/eva_vit_with_tome.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"54a7a818ecf43861"}},{"code_sha256_prefix":"751b1c8ccbe160c4","entry":"cast_if_src_dtype","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/helpers.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"751b1c8ccbe160c4"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/blip2.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"969e8250d805a154","entry":"do_nothing","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/eva_vit_with_tome.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/eva_vit_with_tome.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"969e8250d805a154"}},{"code_sha256_prefix":"432c312da80537d6","entry":"get_clip_timepoints","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/process_video_data.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/process_video_data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"432c312da80537d6"}},{"code_sha256_prefix":"35a743e50e17e66d","entry":"get_sinusoid_encoding_table","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/multimodal_preprocessors.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"35a743e50e17e66d"}},{"code_sha256_prefix":"6bf3a1372bad1076","entry":"kth_bipartite_soft_matching","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/eva_vit_with_tome.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/eva_vit_with_tome.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6bf3a1372bad1076"}},{"code_sha256_prefix":"e1e69d9ba9ffcb24","entry":"load_and_transform_vision_data","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/process_video_data.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/process_video_data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e1e69d9ba9ffcb24"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/modeling_llama.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/base_model.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/base_model.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"71227efe6a7df1d2","entry":"interpolate_pos_encoding","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/multimodal_preprocessors.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"71227efe6a7df1d2"}},{"code_sha256_prefix":"ddd2e52e6b0956b5","entry":"interpolate_pos_encoding_2d","repo":"rese1f/MovieChat","repo_kind":"official","path":"MovieChat/models/multimodal_preprocessors.py","file_url":"https://github.com/rese1f/MovieChat/blob/HEAD/MovieChat/models/multimodal_preprocessors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ddd2e52e6b0956b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}