{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-multi-label-classification","title":"Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer","arxiv_id":"2207.01887","date":"2022-07-05","proceeding":null,"authors":["Sunan He","Taian Guo","Tao Dai","Ruizhi Qiao","Bo Ren","Shu-Tao Xia"],"abstract":"Real-world recognition system often encounters the challenge of unseen labels. To identify such unseen labels, multi-label zero-shot learning (ML-ZSL) focuses on transferring knowledge by a pre-trained textual label embedding (e.g., GloVe). However, such methods only exploit single-modal knowledge from a language model, while ignoring the rich semantic information inherent in image-text pairs. Instead, recently developed open-vocabulary (OV) based methods succeed in exploiting such information of image-text pairs in object detection, and achieve impressive performance. Inspired by the success of OV-based methods, we propose a novel open-vocabulary framework, named multi-modal knowledge transfer (MKT), for multi-label classification. Specifically, our method exploits multi-modal knowledge of image-text pairs based on a vision and language pre-training (VLP) model. To facilitate transferring the image-text matching ability of VLP model, knowledge distillation is employed to guarantee the consistency of image and label embeddings, along with prompt tuning to further update the label embeddings. To further enable the recognition of multiple objects, a simple but effective two-stream module is developed to capture both local and global features. Extensive experimental results show that our method significantly outperforms state-of-the-art methods on public benchmark datasets. The source code is available at https://github.com/sunanhe/MKT.","url_abs":"https://arxiv.org/abs/2207.01887v2","url_pdf":"https://arxiv.org/pdf/2207.01887v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-vocabulary-multi-label-classification","repo_url":"https://github.com/sunanhe/mkt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"multi-label-zero-shot-learning","task_name":"Multi-label zero-shot learning"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multi-label-zero-shot-learning-on-nus-wide","task":"Multi-label zero-shot learning","dataset":"NUS-WIDE","model":"MKT(CLIP)","rank_in_archive_order":1,"of":10,"metrics":{"mAP":"42.7"},"uses_additional_data":false},{"leaderboard":"/sota/multi-label-zero-shot-learning-on-nus-wide","task":"Multi-label zero-shot learning","dataset":"NUS-WIDE","model":"MKT(IN-1K)","rank_in_archive_order":3,"of":10,"metrics":{"mAP":"37.6"},"uses_additional_data":false},{"leaderboard":"/sota/multi-label-zero-shot-learning-on-open-images","task":"Multi-label zero-shot learning","dataset":"Open Images V4","model":"MKT(IN-1K)","rank_in_archive_order":1,"of":8,"metrics":{"MAP":"89.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.01887","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.01887"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sunanhe/mkt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"sunanhe/mkt","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"sunanhe/mkt","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"sunanhe/mkt","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"a97ed48584ab8672","entry":"build_model","repo":"sunanhe/mkt","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a97ed48584ab8672"}},{"code_sha256_prefix":"7b9420cc6794eb72","entry":"load","repo":"sunanhe/mkt","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9420cc6794eb72"}},{"code_sha256_prefix":"5b982b1629319e14","entry":"load_dict","repo":"sunanhe/mkt","repo_kind":"official","path":"utils/dataset.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/utils/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b982b1629319e14"}},{"code_sha256_prefix":"e8fd496a054b64f8","entry":"ranking_lossT","repo":"sunanhe/mkt","repo_kind":"official","path":"models/rank_loss.py","file_url":"https://github.com/sunanhe/mkt/blob/HEAD/models/rank_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8fd496a054b64f8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}