{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crema-multimodal-compositional-video","title":"CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion","arxiv_id":"2402.05889","date":"2024-02-08","proceeding":null,"authors":["Shoubin Yu","Jaehong Yoon","Mohit Bansal"],"abstract":"Despite impressive advancements in recent multimodal reasoning approaches, they are still limited in flexibility and efficiency, as these models typically process only a few fixed modality inputs and require updates to numerous parameters. This paper tackles these critical challenges and proposes CREMA, a generalizable, highly efficient, and modular modality-fusion framework that can incorporate any new modality to enhance video reasoning. We first augment multiple informative modalities (such as optical flow, 3D point cloud, audio, thermal heatmap, and touch map) from given videos without extra human annotation by leveraging sensors or existing pre-trained models. Next, we introduce a query transformer with multiple parameter-efficient modules associated with each accessible modality. It projects diverse modality features to the LLM token embedding space, allowing the model to integrate different data types for response generation. Furthermore, we propose a novel progressive multimodal fusion design supported by a lightweight fusion module and modality-sequential training strategy. It helps compress information across various assisting modalities, maintaining computational efficiency in the LLM while improving performance. We validate our method on 7 video-language reasoning tasks assisted by diverse modalities, including conventional VideoQA and Video-Audio/3D/Touch/Thermal QA, and achieve better/equivalent performance against strong multimodal LLMs, including OneLLM, BLIP-2, and SeViLA while reducing over 90% trainable parameters. We provide extensive analyses of CREMA, including the impact of each modality on reasoning domains, the design of the fusion module, and example visualizations.","url_abs":"https://arxiv.org/abs/2402.05889v3","url_pdf":"https://arxiv.org/pdf/2402.05889v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crema-multimodal-compositional-video","repo_url":"https://github.com/Yui010206/CREMA","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"multimodal-reasoning","task_name":"Multimodal Reasoning"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"response-generation","task_name":"Response Generation"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-sqa3d","task":"Question Answering","dataset":"SQA3D","model":"CREMA","rank_in_archive_order":1,"of":7,"metrics":{"AnswerExactMatch (Question Answering)":"54.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"CREMA","rank_in_archive_order":26,"of":47,"metrics":{"Accuracy":"73.9"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.05889","atlas_url":"https://app.syntology.ai/?focus=2402.05889","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05889"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yui010206/CREMA","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":4,"ran_fixture":1,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"02ce7520da6b7dcc","entry":"HardTopK","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/topk.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/topk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"02ce7520da6b7dcc"}},{"code_sha256_prefix":"0c80b13bc50e20eb","entry":"batched_index_select","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/topk.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/topk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0c80b13bc50e20eb"}},{"code_sha256_prefix":"4fa5290a1f6542b0","entry":"extract_frames_from_indices","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/topk.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/topk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4fa5290a1f6542b0"}},{"code_sha256_prefix":"7d214655954e2bc6","entry":"getAttMap","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/common/gradcam.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/common/gradcam.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7d214655954e2bc6"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/vit.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"98589643273920ba","entry":"main_process","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/common/dist_utils.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"98589643273920ba"}},{"code_sha256_prefix":"791c72070c0b1cfb","entry":"node_to_dict","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/common/config.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/common/config.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"791c72070c0b1cfb"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"643539116e049080","entry":"download_cached_file","repo":"Yui010206/CREMA","repo_kind":"official","path":"lavis/common/dist_utils.py","file_url":"https://github.com/Yui010206/CREMA/blob/HEAD/lavis/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"643539116e049080"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}