{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/get-unlocking-the-multi-modal-potential-of","title":"Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery","arxiv_id":"2403.09974","date":"2024-03-15","proceeding":null,"authors":["Enguang Wang","Zhimao Peng","Zhengyuan Xie","Fei Yang","Xialei Liu","Ming-Ming Cheng"],"abstract":"Given unlabelled datasets containing both old and new categories, generalized category discovery (GCD) aims to accurately discover new classes while correctly classifying old classes, leveraging the class concepts learned from labeled samples. Current GCD methods only use a single visual modality of information, resulting in poor classification of visually similar classes. As a different modality, text information can provide complementary discriminative information, which motivates us to introduce it into the GCD task. However, the lack of class names for unlabelled data makes it impractical to utilize text information. To tackle this challenging problem, in this paper, we propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Specifically, our TES leverages the property that CLIP can generate aligned vision-language features, converting visual embeddings into tokens of the CLIP's text encoder to generate pseudo text embeddings. Besides, we employ a dual-branch framework, through the joint learning and instance consistency of different modality branches, visual and semantic information mutually enhance each other, promoting the interaction and fusion of visual and text knowledge. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks, achieving new state-of-the-art. The code will be released at https://github.com/enguangW/GET .","url_abs":"https://arxiv.org/abs/2403.09974v2","url_pdf":"https://arxiv.org/pdf/2403.09974v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"get-unlocking-the-multi-modal-potential-of","repo_url":"https://github.com/enguangw/get","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"synthesizer","method_name":"Synthesizer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.09974","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.09974"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/enguangw/get","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":5,"ran":2,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"enguangw/get","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"enguangw/get","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"ec22c16f1e0653f7","entry":"get_params_groups","repo":"enguangw/get","repo_kind":"official","path":"model.py","file_url":"https://github.com/enguangw/get/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec22c16f1e0653f7"}},{"code_sha256_prefix":"b3c2e94d4887d26b","entry":"info_nce_logits","repo":"enguangw/get","repo_kind":"official","path":"model.py","file_url":"https://github.com/enguangw/get/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3c2e94d4887d26b"}},{"code_sha256_prefix":"2513029b5bd17df9","entry":"split_cluster_acc_v2","repo":"enguangw/get","repo_kind":"official","path":"util/cluster_and_log_utils.py","file_url":"https://github.com/enguangw/get/blob/HEAD/util/cluster_and_log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2513029b5bd17df9"}},{"code_sha256_prefix":"a291ab390272cf0e","entry":"split_cluster_acc_v2_balanced","repo":"enguangw/get","repo_kind":"official","path":"util/cluster_and_log_utils.py","file_url":"https://github.com/enguangw/get/blob/HEAD/util/cluster_and_log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a291ab390272cf0e"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"enguangw/get","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"42be31a4153f5c2c","entry":"all_sum_item","repo":"enguangw/get","repo_kind":"official","path":"util/cluster_and_log_utils.py","file_url":"https://github.com/enguangw/get/blob/HEAD/util/cluster_and_log_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42be31a4153f5c2c"}},{"code_sha256_prefix":"68683a9ee7224b56","entry":"build_TES_model","repo":"enguangw/get","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68683a9ee7224b56"}},{"code_sha256_prefix":"a059a5361ce98fb0","entry":"build_model","repo":"enguangw/get","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a059a5361ce98fb0"}},{"code_sha256_prefix":"5597baef301aa983","entry":"distill_crit","repo":"enguangw/get","repo_kind":"official","path":"util/general_utils.py","file_url":"https://github.com/enguangw/get/blob/HEAD/util/general_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5597baef301aa983"}},{"code_sha256_prefix":"39511f9d47e3ddc7","entry":"init_experiment","repo":"enguangw/get","repo_kind":"official","path":"util/general_utils.py","file_url":"https://github.com/enguangw/get/blob/HEAD/util/general_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39511f9d47e3ddc7"}},{"code_sha256_prefix":"2da7ec0975be872a","entry":"load","repo":"enguangw/get","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2da7ec0975be872a"}},{"code_sha256_prefix":"bd0cff7a90b9726a","entry":"load_TES_CLIP","repo":"enguangw/get","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/enguangw/get/blob/HEAD/clip/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd0cff7a90b9726a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}