{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficientvlm-fast-and-accurate-vision","title":"EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning","arxiv_id":"2210.07795","date":"2022-10-14","proceeding":null,"authors":["Tiannan Wang","Wangchunshu Zhou","Yan Zeng","Xinsong Zhang"],"abstract":"Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and latency constraints. In this work, we introduce a distilling then pruning framework to compress large vision-language models into smaller, faster, and more accurate ones. We first shrink the size of a pre-trained large VLM and apply knowledge distillation in the vision-language pre-training stage to obtain a task-agnostic compact VLM. Then we propose a modal-adaptive pruning algorithm to automatically infer the importance of vision and language modalities for different downstream tasks and adaptively remove redundant structures and neurons in different encoders with controllable target sparsity. We apply our framework to train EfficientVLM, a fast and accurate vision-language model consisting of 6 vision layers, 3 text layers, and 3 cross-modal fusion layers, accounting for only 93 million parameters in total, which is 44.3% of the teacher model. EfficientVLM retains 98.4% performance of the teacher model and accelerates its inference speed by 2.2x. EfficientVLM achieves a large absolute improvement over previous SoTA efficient VLMs of similar sizes by a large margin on various vision-language tasks, including VQAv2 (+4.9%), NLVR2 (+5.6%), ITR (R@1 on TR +17.2%, on IR + 15.6% ) and COCO caption generation (CIDEr +6.5), demonstrating a large potential on training lightweight VLMs.","url_abs":"https://arxiv.org/abs/2210.07795v1","url_pdf":"https://arxiv.org/pdf/2210.07795v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficientvlm-fast-and-accurate-vision","repo_url":"https://github.com/swaggy-tn/efficientvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2210.07795","atlas_url":"https://app.syntology.ai/?focus=2210.07795","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07795"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/swaggy-tn/efficientvlm","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"e5b89b1f72c319ed","entry":"clip_loss","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"efficient_models/eff_vit.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/efficient_models/eff_vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e5b89b1f72c319ed"}},{"code_sha256_prefix":"4675779aec92d6c0","entry":"contrastive_loss","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"efficient_models/eff_vit.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/efficient_models/eff_vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"4675779aec92d6c0"}},{"code_sha256_prefix":"a59f653ef4653ae4","entry":"get_cor_teacher","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"Eff_Captioning.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/Eff_Captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"a59f653ef4653ae4"}},{"code_sha256_prefix":"a6e8b08bf21e4ea6","entry":"get_kd_loss","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"Eff_Captioning.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/Eff_Captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"a6e8b08bf21e4ea6"}},{"code_sha256_prefix":"e9a9fd7c33b84f9a","entry":"soft_cross_entropy","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"Eff_Captioning.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/Eff_Captioning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e9a9fd7c33b84f9a"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"efficient_models/model_generation.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/efficient_models/model_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"9e8f81565e828ebf","entry":"top_k_top_p_filtering","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"efficient_models/eff_bert.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/efficient_models/eff_bert.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"9e8f81565e828ebf"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"efficient_models/eff_bert.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/efficient_models/eff_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}},{"code_sha256_prefix":"e3d6fbf640a143ee","entry":"reinit_scheduler_properties_mysched","repo":"swaggy-tn/efficientvlm","repo_kind":"official","path":"Captioning_pretrain.py","file_url":"https://github.com/swaggy-tn/efficientvlm/blob/HEAD/Captioning_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e3d6fbf640a143ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}