{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/matvlm-hybrid-mamba-transformer-for-efficient","title":"MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling","arxiv_id":"2503.13440","date":"2025-03-17","proceeding":null,"authors":["Yingyue Li","Bencheng Liao","Wenyu Liu","Xinggang Wang"],"abstract":"With the advancement of RNN models with linear complexity, the quadratic complexity challenge of transformers has the potential to be overcome. Notably, the emerging Mamba-2 has demonstrated competitive performance, bridging the gap between RNN models and transformers. However, due to sequential processing and vanishing gradients, RNN models struggle to capture long-range dependencies, limiting contextual understanding. This results in slow convergence, high resource demands, and poor performance on downstream understanding and complex reasoning tasks. In this work, we present a hybrid model MaTVLM by substituting a portion of the transformer decoder layers in a pre-trained VLM with Mamba-2 layers. Leveraging the inherent relationship between attention and Mamba-2, we initialize Mamba-2 with corresponding attention weights to accelerate convergence. Subsequently, we employ a single-stage distillation process, using the pre-trained VLM as the teacher model to transfer knowledge to the MaTVLM, further enhancing convergence speed and performance. Furthermore, we investigate the impact of differential distillation loss within our training framework. We evaluate the MaTVLM on multiple benchmarks, demonstrating competitive performance against the teacher model and existing VLMs while surpassing both Mamba-based VLMs and models of comparable parameter scales. Remarkably, the MaTVLM achieves up to 3.6x faster inference than the teacher model while reducing GPU memory consumption by 27.5%, all without compromising performance. Code and models are released at http://github.com/hustvl/MaTVLM.","url_abs":"https://arxiv.org/abs/2503.13440v1","url_pdf":"https://arxiv.org/pdf/2503.13440v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"matvlm-hybrid-mamba-transformer-for-efficient","repo_url":"https://github.com/hustvl/MaTVLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.13440","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.13440"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hustvl/MaTVLM","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"hustvl/MaTVLM","repo_kind":"official","path":"mamba2/hybrid_mamba_layer.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/mamba2/hybrid_mamba_layer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"hustvl/MaTVLM","repo_kind":"official","path":"tinyllava/eval/model_vqa_loader.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/tinyllava/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"e845d6cad3dda7eb","entry":"construct_layer_dict","repo":"hustvl/MaTVLM","repo_kind":"official","path":"util.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e845d6cad3dda7eb"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"hustvl/MaTVLM","repo_kind":"official","path":"tinyllava/eval/model_vqa.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/tinyllava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"72afeaae843bd353","entry":"load_safetensors_to_dict","repo":"hustvl/MaTVLM","repo_kind":"official","path":"util.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72afeaae843bd353"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"hustvl/MaTVLM","repo_kind":"official","path":"tinyllava/eval/model_vqa.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/tinyllava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"0d94ed5522773864","entry":"construct_language_layer_dict","repo":"hustvl/MaTVLM","repo_kind":"official","path":"util.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d94ed5522773864"}},{"code_sha256_prefix":"4ada4f4ed6ff4fec","entry":"decode","repo":"hustvl/MaTVLM","repo_kind":"official","path":"mamba2_inference/hybrid_wrapper_tinyllava.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/mamba2_inference/hybrid_wrapper_tinyllava.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ada4f4ed6ff4fec"}},{"code_sha256_prefix":"afef156338836ef8","entry":"merge_projections_for_layers","repo":"hustvl/MaTVLM","repo_kind":"official","path":"mamba2_inference/hybrid_wrapper.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/mamba2_inference/hybrid_wrapper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afef156338836ef8"}},{"code_sha256_prefix":"1f13fabd04c36b6a","entry":"merge_projections_for_layers","repo":"hustvl/MaTVLM","repo_kind":"official","path":"mamba2_inference/hybrid_wrapper_tinyllava.py","file_url":"https://github.com/hustvl/MaTVLM/blob/HEAD/mamba2_inference/hybrid_wrapper_tinyllava.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f13fabd04c36b6a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}