{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/textual-knowledge-matters-cross-modality-co","title":"Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery","arxiv_id":"2403.07369","date":"2024-03-12","proceeding":null,"authors":["Haiyang Zheng","Nan Pu","Wenjing Li","Nicu Sebe","Zhun Zhong"],"abstract":"In this paper, we study the problem of Generalized Category Discovery (GCD), which aims to cluster unlabeled data from both known and unknown categories using the knowledge of labeled data from known categories. Current GCD methods rely on only visual cues, which however neglect the multi-modality perceptive nature of human cognitive processes in discovering novel visual categories. To address this, we propose a two-phase TextGCD framework to accomplish multi-modality GCD by exploiting powerful Visual-Language Models. TextGCD mainly includes a retrieval-based text generation (RTG) phase and a cross-modality co-teaching (CCT) phase. First, RTG constructs a visual lexicon using category tags from diverse datasets and attributes from Large Language Models, generating descriptive texts for images in a retrieval manner. Second, CCT leverages disparities between textual and visual modalities to foster mutual learning, thereby enhancing visual GCD. In addition, we design an adaptive class aligning strategy to ensure the alignment of category perceptions between modalities as well as a soft-voting mechanism to integrate multi-modality cues. Experiments on eight datasets show the large superiority of our approach over state-of-the-art methods. Notably, our approach outperforms the best competitor, by 7.7% and 10.8% in All accuracy on ImageNet-1k and CUB, respectively.","url_abs":"https://arxiv.org/abs/2403.07369v2","url_pdf":"https://arxiv.org/pdf/2403.07369v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"textual-knowledge-matters-cross-modality-co","repo_url":"https://github.com/haiyangzheng/textgcd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cct","method_name":"CCT"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.07369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haiyangzheng/textgcd","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"f8466b7c87d6147d","entry":"process_file","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/text_process.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/text_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8466b7c87d6147d"}},{"code_sha256_prefix":"8bd0b2a28f64b67a","entry":"set_experiment_directories","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/general_utils.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/general_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bd0b2a28f64b67a"}},{"code_sha256_prefix":"b14783f90aebfa6a","entry":"set_logging","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/general_utils.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/general_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b14783f90aebfa6a"}},{"code_sha256_prefix":"7ff96fb808701248","entry":"simgcd_distillLoss","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/train_utils.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ff96fb808701248"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"c47aa9e5b049a11d","entry":"build_model","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c47aa9e5b049a11d"}},{"code_sha256_prefix":"7e356a69c445291a","entry":"image_text_contrastive_loss","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/train_utils.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e356a69c445291a"}},{"code_sha256_prefix":"aebda4e166ad65b3","entry":"load","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/clip/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aebda4e166ad65b3"}},{"code_sha256_prefix":"00bcaa100d09df79","entry":"select_confident_samples","repo":"haiyangzheng/textgcd","repo_kind":"official","path":"utils/train_utils.py","file_url":"https://github.com/haiyangzheng/textgcd/blob/HEAD/utils/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00bcaa100d09df79"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}