{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-multimodal-learning-from-data","title":"Efficient Multimodal Learning from Data-centric Perspective","arxiv_id":"2402.11530","date":"2024-02-18","proceeding":null,"authors":["Muyang He","Yexin Liu","Boya Wu","Jianhao Yuan","Yueze Wang","Tiejun Huang","Bo Zhao"],"abstract":"Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks. However, their deployment is hindered by substantial computational costs in both training and inference, limiting accessibility to the broader research and user communities. A straightforward solution is to leverage smaller pre-trained vision and language models, which inevitably cause significant performance drops. In this paper, we demonstrate the possibility of training a smaller but better MLLM with high-quality training data. Specifically, we introduce Bunny, a family of lightweight MLLMs with flexible vision and language backbones for efficient multimodal learning from selected training data. Experiments show that our Bunny-4B/8B outperforms the state-of-the-art large MLLMs on multiple benchmarks. We expect that this work can provide the community with a clean and flexible open-source tool for further research and development. The code, models, and data can be found in https://github.com/BAAI-DCAI/Bunny.","url_abs":"https://arxiv.org/abs/2402.11530v3","url_pdf":"https://arxiv.org/pdf/2402.11530v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-multimodal-learning-from-data","repo_url":"https://github.com/baai-dcai/bunny","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-generation","task_name":"Referring expression generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-coloninst-v1-seen","task":"Image Classification","dataset":"ColonINST-v1 (Seen)","model":"Bunny-v1.0-3B (w/ LoRA, w/ extra data)","rank_in_archive_order":11,"of":17,"metrics":{"Accuray":"92.47"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-seen","task":"Image Classification","dataset":"ColonINST-v1 (Seen)","model":"Bunny-v1.0-3B (w/ LoRA, w/o extra data)","rank_in_archive_order":13,"of":17,"metrics":{"Accuray":"91.16"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-unseen","task":"Image Classification","dataset":"ColonINST-v1 (Unseen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":4,"of":17,"metrics":{"Accuray":"79.50"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-unseen","task":"Image Classification","dataset":"ColonINST-v1 (Unseen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":14,"of":17,"metrics":{"Accuray":"75.50"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst","task":"Referring expression generation","dataset":"ColonINST-v1 (Seen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":11,"of":17,"metrics":{"Accuray":"96.61"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst","task":"Referring expression generation","dataset":"ColonINST-v1 (Seen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":12,"of":17,"metrics":{"Accuray":"96.02"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst-1","task":"Referring expression generation","dataset":"ColonINST-v1 (Unseen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":4,"of":17,"metrics":{"Accuray":"75.08"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst-1","task":"Referring expression generation","dataset":"ColonINST-v1 (Unseen)","model":"Bunny-v1.0-3B\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":15,"of":17,"metrics":{"Accuray":"69.45"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.11530","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11530"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baai-dcai/bunny","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2fd1673b298c086c","entry":"construct_prompt","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa_cmmmu.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa_cmmmu.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fd1673b298c086c"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa_mmbench.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"8bc914474691d84a","entry":"load_pretrained_model","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/model/builder.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/model/builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bc914474691d84a"}},{"code_sha256_prefix":"ce93b7e192e1a8b7","entry":"load_yaml","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa_cmmmu.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa_cmmmu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce93b7e192e1a8b7"}},{"code_sha256_prefix":"f3018307f9301da1","entry":"parse_multi_choice_response","repo":"baai-dcai/bunny","repo_kind":"official","path":"bunny/eval/model_vqa_mmmu.py","file_url":"https://github.com/baai-dcai/bunny/blob/HEAD/bunny/eval/model_vqa_mmmu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3018307f9301da1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}