{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dense-connector-for-mllms","title":"Dense Connector for MLLMs","arxiv_id":"2405.13800","date":"2024-05-22","proceeding":null,"authors":["Huanjin Yao","Wenhao Wu","Taojiannan Yang","Yuxin Song","Mengxi Zhang","Haocheng Feng","Yifan Sun","Zhiheng Li","Wanli Ouyang","Jingdong Wang"],"abstract":"Do we fully leverage the potential of visual encoder in Multimodal Large Language Models (MLLMs)? The recent outstanding performance of MLLMs in multimodal understanding has garnered broad attention from both academia and industry. In the current MLLM rat race, the focus seems to be predominantly on the linguistic side. We witness the rise of larger and higher-quality instruction datasets, as well as the involvement of larger-sized LLMs. Yet, scant attention has been directed towards the visual signals utilized by MLLMs, often assumed to be the final high-level features extracted by a frozen visual encoder. In this paper, we introduce the Dense Connector - a simple, effective, and plug-and-play vision-language connector that significantly enhances existing MLLMs by leveraging multi-layer visual features, with minimal additional computational overhead. Building on this, we also propose the Efficient Dense Connector, which achieves performance comparable to LLaVA-v1.5 with only 25% of the visual tokens. Furthermore, our model, trained solely on images, showcases remarkable zero-shot capabilities in video understanding as well. Experimental results across various vision encoders, image resolutions, training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse architectures of MLLMs (e.g., LLaVA-v1.5, LLaVA-NeXT and Mini-Gemini) validate the versatility and scalability of our approach, achieving state-of-the-art performance across 19 image and video benchmarks. We hope that this work will provide valuable experience and serve as a basic module for future MLLM development. Code is available at https://github.com/HJYao00/DenseConnector .","url_abs":"https://arxiv.org/abs/2405.13800v2","url_pdf":"https://arxiv.org/pdf/2405.13800v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dense-connector-for-mllms","repo_url":"https://github.com/HJYao00/DenseConnector","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13800","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13800"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HJYao00/DenseConnector","reach":null}],"summary":{"ran_fixture":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"be57d00e5f80aa2e","entry":"dense_connector_dci","repo":"HJYao00/DenseConnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/HJYao00/DenseConnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be57d00e5f80aa2e"}},{"code_sha256_prefix":"14a1bc132ea59a9f","entry":"dense_connector_dci","repo":"hjyao00/denseconnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/hjyao00/denseconnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14a1bc132ea59a9f"}},{"code_sha256_prefix":"ba53c93945b602f4","entry":"dense_connector_sci","repo":"HJYao00/DenseConnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/HJYao00/DenseConnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba53c93945b602f4"}},{"code_sha256_prefix":"24ed3fa200b261dc","entry":"dense_connector_sci","repo":"hjyao00/denseconnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/hjyao00/denseconnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24ed3fa200b261dc"}},{"code_sha256_prefix":"03b91d218209f7f9","entry":"dense_connector_sti","repo":"HJYao00/DenseConnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/HJYao00/DenseConnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03b91d218209f7f9"}},{"code_sha256_prefix":"cc348531d44c97af","entry":"dense_connector","repo":"HJYao00/DenseConnector","repo_kind":"official","path":"dc/model/multimodal_projector/dense_connector.py","file_url":"https://github.com/HJYao00/DenseConnector/blob/HEAD/dc/model/multimodal_projector/dense_connector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc348531d44c97af"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}