{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/moextend-tuning-new-experts-for-modality-and","title":"MoExtend: Tuning New Experts for Modality and Task Extension","arxiv_id":"2408.03511","date":"2024-08-07","proceeding":null,"authors":["Shanshan Zhong","ShangHua Gao","Zhongzhan Huang","Wushao Wen","Marinka Zitnik","Pan Zhou"],"abstract":"Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research. Code: https://github.com/zhongshsh/MoExtend.","url_abs":"https://arxiv.org/abs/2408.03511v1","url_pdf":"https://arxiv.org/pdf/2408.03511v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"moextend-tuning-new-experts-for-modality-and","repo_url":"https://github.com/zhongshsh/moextend","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.03511","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.03511"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zhongshsh/MoExtend","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f251388ba91a580d","entry":"MixtralConfig","repo":"zhongshsh/MoExtend","repo_kind":"official","path":"src/model_moextend.py","file_url":"https://github.com/zhongshsh/MoExtend/blob/HEAD/src/model_moextend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f251388ba91a580d"}},{"code_sha256_prefix":"0983b4b1438b8ac2","entry":"MixtralSparseMoeBlock","repo":"zhongshsh/MoExtend","repo_kind":"official","path":"src/model_moextend.py","file_url":"https://github.com/zhongshsh/MoExtend/blob/HEAD/src/model_moextend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0983b4b1438b8ac2"}},{"code_sha256_prefix":"60cbc5cb83d64366","entry":"WeightScale","repo":"zhongshsh/MoExtend","repo_kind":"official","path":"src/model_moextend.py","file_url":"https://github.com/zhongshsh/MoExtend/blob/HEAD/src/model_moextend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60cbc5cb83d64366"}},{"code_sha256_prefix":"2f745d254ea9ba37","entry":"MixtralBLockSparseTop2MLP","repo":"zhongshsh/MoExtend","repo_kind":"official","path":"src/model_moextend.py","file_url":"https://github.com/zhongshsh/MoExtend/blob/HEAD/src/model_moextend.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f745d254ea9ba37"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}