{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llava-kd-a-framework-of-distilling-multimodal","title":"LLaVA-KD: A Framework of Distilling Multimodal Large Language Models","arxiv_id":"2410.16236","date":"2024-10-21","proceeding":null,"authors":["Yuxuan Cai","Jiangning Zhang","Haoyang He","Xinwei He","Ao Tong","Zhenye Gan","Chengjie Wang","Xiang Bai"],"abstract":"The success of Large Language Models (LLM) has led researchers to explore Multimodal Large Language Models (MLLM) for unified visual and linguistic understanding. However, the increasing model size and computational complexity of MLLM limit their use in resource-constrained environments. Small-scale MLLM (s-MLLM) aims to retain the capabilities of the large-scale model (l-MLLM) while reducing computational demands, but resulting in a significant decline in performance. To address the aforementioned issues, we propose a novel LLaVA-KD framework to transfer knowledge from l-MLLM to s-MLLM. Specifically, we introduce Multimodal Distillation (MDist) to minimize the divergence between the visual-textual output distributions of l-MLLM and s-MLLM, and Relation Distillation (RDist) to transfer l-MLLM's ability to model correlations between visual features. Additionally, we propose a three-stage training scheme to fully exploit the potential of s-MLLM: 1) Distilled Pre-Training to align visual-textual representations, 2) Supervised Fine-Tuning to equip the model with multimodal understanding, and 3) Distilled Fine-Tuning to further transfer l-MLLM capabilities. Our approach significantly improves performance without altering the small model's architecture. Extensive experiments and ablation studies validate the effectiveness of each proposed component. Code will be available at https://github.com/Fantasyele/LLaVA-KD.","url_abs":"https://arxiv.org/abs/2410.16236v2","url_pdf":"https://arxiv.org/pdf/2410.16236v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llava-kd-a-framework-of-distilling-multimodal","repo_url":"https://github.com/Fantasyele/LLaVA-KD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.16236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.16236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Fantasyele/LLaVA-KD","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":5,"ran_violates":1,"unverified":8},"by_repo_kind":{"official":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/model_vqa_loader.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/model_vqa.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"c2cb10d944648763","entry":"get_value_from_kwargs","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/model/modeling_LLaVA_KD.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/model/modeling_LLaVA_KD.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2cb10d944648763"}},{"code_sha256_prefix":"a7bee88c1c7fd6a3","entry":"image_parser","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/run_tiny_llava.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/run_tiny_llava.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7bee88c1c7fd6a3"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/model_vqa_mmbench.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/model_vqa.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"eb488a6c509bb0b2","entry":"forward_kl","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/utils/distill_loss_utils.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/utils/distill_loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb488a6c509bb0b2"}},{"code_sha256_prefix":"391fc0746cff81a2","entry":"load_base_ckp_for_lora","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/model/load_model.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/model/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"391fc0746cff81a2"}},{"code_sha256_prefix":"4552e857b0cf7ef7","entry":"load_image","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/run_tiny_llava.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/run_tiny_llava.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4552e857b0cf7ef7"}},{"code_sha256_prefix":"99d66c86efac69ff","entry":"load_images","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/run_tiny_llava.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/run_tiny_llava.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99d66c86efac69ff"}},{"code_sha256_prefix":"851298b43d95d91e","entry":"load_pretrained_model","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/model/load_model.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/model/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"851298b43d95d91e"}},{"code_sha256_prefix":"f3018307f9301da1","entry":"parse_multi_choice_response","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/eval/model_vqa_mmmu.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/eval/model_vqa_mmmu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3018307f9301da1"}},{"code_sha256_prefix":"badd0b27a608c8c8","entry":"reverse_kl","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/utils/distill_loss_utils.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/utils/distill_loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"badd0b27a608c8c8"}},{"code_sha256_prefix":"38f61e23457096f4","entry":"symmetric_kl","repo":"Fantasyele/LLaVA-KD","repo_kind":"official","path":"llavakd/utils/distill_loss_utils.py","file_url":"https://github.com/Fantasyele/LLaVA-KD/blob/HEAD/llavakd/utils/distill_loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38f61e23457096f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}