{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leveraging-cross-modal-neighbor","title":"Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification","arxiv_id":"2404.17753","date":"2024-04-27","proceeding":"CVPR 2024 1","authors":["Chao Yi","Lu Ren","De-Chuan Zhan","Han-Jia Ye"],"abstract":"CLIP showcases exceptional cross-modal matching capabilities due to its training on image-text contrastive learning tasks. However, without specific optimization for unimodal scenarios, its performance in single-modality feature extraction might be suboptimal. Despite this, some studies have directly used CLIP's image encoder for tasks like few-shot classification, introducing a misalignment between its pre-training objectives and feature extraction methods. This inconsistency can diminish the quality of the image's feature representation, adversely affecting CLIP's effectiveness in target tasks. In this paper, we view text features as precise neighbors of image features in CLIP's space and present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. This feature extraction method aligns better with CLIP's pre-training objectives, thereby fully leveraging CLIP's robust cross-modal capabilities. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. We introduce the Auto Text Generator(ATG) to automatically generate the required texts in a data-free and training-free manner. We apply CODER to CLIP's zero-shot and few-shot image classification tasks. Experiment results across various datasets and models confirm CODER's effectiveness. Code is available at:https://github.com/YCaigogogo/CVPR24-CODER.","url_abs":"https://arxiv.org/abs/2404.17753v1","url_pdf":"https://arxiv.org/pdf/2404.17753v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leveraging-cross-modal-neighbor","repo_url":"https://github.com/ycaigogogo/cvpr24-coder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"few-shot-image-classification","task_name":"Few-Shot Image Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.17753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.17753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ycaigogogo/cvpr24-coder","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/YCaigogogo/CVPR24-CODER","reach":{"status":"ok"}}],"summary":{"ran":9,"ran_draft_wrong":1,"ran_honours":1,"unverified":9},"by_repo_kind":{"official":{"samples":20,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":20,"samples":[{"code_sha256_prefix":"cd842675cb581de9","entry":"get_abs_path","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"datasets/dataset_config.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/datasets/dataset_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd842675cb581de9"}},{"code_sha256_prefix":"a64d39d1a99f8f42","entry":"get_ana_templates","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a64d39d1a99f8f42"}},{"code_sha256_prefix":"979ae65cc828c9e4","entry":"get_att_templates","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"979ae65cc828c9e4"}},{"code_sha256_prefix":"644c2a9c22e5f2f2","entry":"get_class_names","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_zero_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"644c2a9c22e5f2f2"}},{"code_sha256_prefix":"86c82affa2d154f9","entry":"get_name_templates","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86c82affa2d154f9"}},{"code_sha256_prefix":"51bee0de7e56c2ea","entry":"get_ovo_templates","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51bee0de7e56c2ea"}},{"code_sha256_prefix":"1d5e24ac93408ca4","entry":"heuristic_classifier","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_zero_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d5e24ac93408ca4"}},{"code_sha256_prefix":"d9847cc2f8495b76","entry":"load_pickle","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"datasets/dataset_class.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/datasets/dataset_class.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9847cc2f8495b76"}},{"code_sha256_prefix":"d94bc26deebfbac5","entry":"minmax_normalization","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d94bc26deebfbac5"}},{"code_sha256_prefix":"f321f54723433661","entry":"pil_loader","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"datasets/dataset_class.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/datasets/dataset_class.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f321f54723433661"}},{"code_sha256_prefix":"9b9ce4000929d33f","entry":"update_cfg","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b9ce4000929d33f"}},{"code_sha256_prefix":"cdb65597626e1418","entry":"CODER","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdb65597626e1418"}},{"code_sha256_prefix":"39e6b23b55f376ea","entry":"build_model","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"CLIP/clip/model.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/CLIP/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39e6b23b55f376ea"}},{"code_sha256_prefix":"f2a0a5061e7c204e","entry":"cls_acc","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2a0a5061e7c204e"}},{"code_sha256_prefix":"6ee57d74d436ecab","entry":"get_cache","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_few_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ee57d74d436ecab"}},{"code_sha256_prefix":"5e8258eccc2a7bfa","entry":"get_templates","repo":"YCaigogogo/CVPR24-CODER","repo_kind":"official","path":"CODER/CODER.py","file_url":"https://github.com/YCaigogogo/CVPR24-CODER/blob/HEAD/CODER/CODER.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e8258eccc2a7bfa"}},{"code_sha256_prefix":"669c0f6e8ba13fa3","entry":"get_templates","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"CODER/template.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/CODER/template.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"669c0f6e8ba13fa3"}},{"code_sha256_prefix":"216a738551d825eb","entry":"get_test_feature","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_few_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"216a738551d825eb"}},{"code_sha256_prefix":"558e0e421031aff9","entry":"get_val_feature","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_few_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"558e0e421031aff9"}},{"code_sha256_prefix":"5e6389c00af848a2","entry":"modify_top_gap","repo":"ycaigogogo/cvpr24-coder","repo_kind":"official","path":"classify_zero_shot.py","file_url":"https://github.com/ycaigogogo/cvpr24-coder/blob/HEAD/classify_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e6389c00af848a2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}