{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/move-kd-knowledge-distillation-for-vlms-with","title":"MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders","arxiv_id":"2501.01709","date":"2025-01-03","proceeding":"CVPR 2025 1","authors":["Jiajun Cao","Yuan Zhang","Tao Huang","Ming Lu","Qizhe Zhang","Ruichuan An","Ningning Ma","Shanghang Zhang"],"abstract":"Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different visual encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. The code will be released.","url_abs":"https://arxiv.org/abs/2501.01709v1","url_pdf":"https://arxiv.org/pdf/2501.01709v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.01709","atlas_url":"https://app.syntology.ai/?focus=2501.01709","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.01709"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hey-cjj/MoVE-KD","reach":null}],"summary":{"ran":2},"by_repo_kind":{"found_in_text":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9350c4a5140680cc","entry":"LoRALayer","repo":"hey-cjj/MoVE-KD","repo_kind":"found_in_text","path":"llava/model/multimodal_encoder/moe_lora.py","file_url":"https://github.com/hey-cjj/MoVE-KD/blob/HEAD/llava/model/multimodal_encoder/moe_lora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9350c4a5140680cc"}},{"code_sha256_prefix":"51ed776598ad1795","entry":"MoE_LoRA_CLIP","repo":"hey-cjj/MoVE-KD","repo_kind":"found_in_text","path":"llava/model/multimodal_encoder/moe_lora.py","file_url":"https://github.com/hey-cjj/MoVE-KD/blob/HEAD/llava/model/multimodal_encoder/moe_lora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51ed776598ad1795"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}