{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/k-lite-learning-transferable-visual-models","title":"K-LITE: Learning Transferable Visual Models with External Knowledge","arxiv_id":"2204.09222","date":"2022-04-20","proceeding":null,"authors":["Sheng Shen","Chunyuan Li","Xiaowei Hu","Jianwei Yang","Yujia Xie","Pengchuan Zhang","Zhe Gan","Lijuan Wang","Lu Yuan","Ce Liu","Kurt Keutzer","Trevor Darrell","Anna Rohrbach","Jianfeng Gao"],"abstract":"The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at https://github.com/microsoft/klite.","url_abs":"https://arxiv.org/abs/2204.09222v2","url_pdf":"https://arxiv.org/pdf/2204.09222v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"k-lite-learning-transferable-visual-models","repo_url":"https://github.com/microsoft/klite","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"k-lite-learning-transferable-visual-models","repo_url":"https://github.com/microsoft/unicl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.09222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.09222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/klite","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/unicl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_violates":1,"unverified":5},"by_repo_kind":{"listed":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5913171a402effd1","entry":"PatchEmbed","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5913171a402effd1"}},{"code_sha256_prefix":"89e08495ccdd55f4","entry":"PatchMerging","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89e08495ccdd55f4"}},{"code_sha256_prefix":"5a5551bec22c5f90","entry":"SwinTransformerBlock","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a5551bec22c5f90"}},{"code_sha256_prefix":"44721c30a2563ec1","entry":"WindowAttention","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44721c30a2563ec1"}},{"code_sha256_prefix":"253f375438a3b31d","entry":"is_lang_encoder","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"253f375438a3b31d"}},{"code_sha256_prefix":"30fb736aaa42b89d","entry":"BasicLayer","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30fb736aaa42b89d"}},{"code_sha256_prefix":"96a6b788f6e04b42","entry":"KLITEModel","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96a6b788f6e04b42"}},{"code_sha256_prefix":"5817ae83cc9df84f","entry":"SwinTransformer","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5817ae83cc9df84f"}},{"code_sha256_prefix":"d7ee228866431711","entry":"build_tokenizer","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7ee228866431711"}},{"code_sha256_prefix":"25812f971a96b8f8","entry":"lang_encoders","repo":"microsoft/klite","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/microsoft/klite/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25812f971a96b8f8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}