{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/align-kd-distilling-cross-modal-alignment","title":"Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model","arxiv_id":"2412.01282","date":"2024-12-02","proceeding":null,"authors":["Qianhan Feng","Wenshuo Li","Tong Lin","Xinghao Chen"],"abstract":"Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks. Meanwhile, the great need for capable aritificial intelligence on mobile devices also arises, such as the AI assistant software. Some efforts try to migrate VLMs to edge devices to expand their application scope. Simplifying the model structure is a common method, but as the model shrinks, the trade-off between performance and size becomes more and more difficult. Knowledge distillation (KD) can help models improve comprehensive capabilities without increasing size or data volume. However, most of the existing large model distillation techniques only consider applications on single-modal LLMs, or only use teachers to create new data environments for students. None of these methods take into account the distillation of the most important cross-modal alignment knowledge in VLMs. We propose a method called Align-KD to guide the student model to learn the cross-modal matching that occurs at the shallow layer. The teacher also helps student learn the projection of vision token into text embedding space based on the focus of text. Under the guidance of Align-KD, the 1.7B MobileVLM V2 model can learn rich knowledge from the 7B teacher model with light design of training loss, and achieve an average score improvement of 2.0 across 6 benchmarks under two training subsets respectively. Code is available at: https://github.com/fqhank/Align-KD.","url_abs":"https://arxiv.org/abs/2412.01282v1","url_pdf":"https://arxiv.org/pdf/2412.01282v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"align-kd-distilling-cross-modal-alignment","repo_url":"https://github.com/fqhank/align-kd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.01282","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.01282"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fqhank/align-kd","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"fqhank/align-kd","repo_kind":"official","path":"mobilevlm/train/trainer.py","file_url":"https://github.com/fqhank/align-kd/blob/HEAD/mobilevlm/train/trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"fac53f6bae1b69a4","entry":"get_length_grouped_indices","repo":"fqhank/align-kd","repo_kind":"official","path":"mobilevlm/train/trainer.py","file_url":"https://github.com/fqhank/align-kd/blob/HEAD/mobilevlm/train/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fac53f6bae1b69a4"}},{"code_sha256_prefix":"d61903181e150045","entry":"get_modality_length_grouped_indices","repo":"fqhank/align-kd","repo_kind":"official","path":"mobilevlm/train/trainer.py","file_url":"https://github.com/fqhank/align-kd/blob/HEAD/mobilevlm/train/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61903181e150045"}},{"code_sha256_prefix":"ddb56d717206dc41","entry":"load_pretrained_model_processor","repo":"fqhank/align-kd","repo_kind":"official","path":"mobilevlm/model/mobilevlm.py","file_url":"https://github.com/fqhank/align-kd/blob/HEAD/mobilevlm/model/mobilevlm.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddb56d717206dc41"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}