{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cumo-scaling-multimodal-llm-with-co-upcycled","title":"CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts","arxiv_id":"2405.05949","date":"2024-05-09","proceeding":null,"authors":["Jiachen Li","Xinyao Wang","Sijie Zhu","Chia-Wen Kuo","Lu Xu","Fan Chen","Jitesh Jain","Humphrey Shi","Longyin Wen"],"abstract":"Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.","url_abs":"https://arxiv.org/abs/2405.05949v1","url_pdf":"https://arxiv.org/pdf/2405.05949v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cumo-scaling-multimodal-llm-with-co-upcycled","repo_url":"https://github.com/shi-labs/cumo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-instruction-following","task_name":"visual instruction following"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"CuMo-7B","rank_in_archive_order":60,"of":231,"metrics":{"GPT-4 score":"51.0","Params":"7B"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mmbench","task":"Visual Question Answering","dataset":"MMBench","model":"CuMo-7B","rank_in_archive_order":2,"of":5,"metrics":{"GPT-3.5 score":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-gqa-test-dev","task":"Visual Question Answering (VQA)","dataset":"GQA test-dev","model":"CuMo-7B","rank_in_archive_order":3,"of":17,"metrics":{"Accuracy":"64.9"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"CuMo-7B","rank_in_archive_order":6,"of":56,"metrics":{"Accuracy":"82.2"},"uses_additional_data":true},{"leaderboard":"/sota/visual-instruction-following-on-llava-bench","task":"visual instruction following","dataset":"LLaVA-Bench","model":"CuMo-7B","rank_in_archive_order":1,"of":8,"metrics":{"avg score":"85.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.05949","atlas_url":"https://app.syntology.ai/?focus=2405.05949","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.05949"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shi-labs/cumo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran":4,"ran_violates":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa_loader.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/mm_utils.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa_mmbench.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"ff5aa85ec44db5ee","entry":"normalize_extracted_answer","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/calculate_score.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/calculate_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff5aa85ec44db5ee"}},{"code_sha256_prefix":"d6888be4b2989f1f","entry":"process_single_sample","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa_mathvista.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa_mathvista.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6888be4b2989f1f"}},{"code_sha256_prefix":"9ffa907555068ff3","entry":"read_json","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/calculate_score.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/calculate_score.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ffa907555068ff3"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/mm_utils.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/mm_utils.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"ed80aaedae521e4f","entry":"construct_prompt","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/model_vqa_mathvista.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/model_vqa_mathvista.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed80aaedae521e4f"}},{"code_sha256_prefix":"c28eb86a8d14e4ed","entry":"get_most_similar","repo":"shi-labs/cumo","repo_kind":"official","path":"cumo/eval/calculate_score.py","file_url":"https://github.com/shi-labs/cumo/blob/HEAD/cumo/eval/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c28eb86a8d14e4ed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}