{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-text-cross-alignment-refining-the","title":"Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models","arxiv_id":"2406.02915","date":"2024-06-05","proceeding":null,"authors":["Jinhao Li","Haopeng Li","Sarah Erfani","Lei Feng","James Bailey","Feng Liu"],"abstract":"It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions generated by a large language model can significantly enhance zero-shot performance. However, in this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image, and then we theoretically validate this finding. Thus, we present a method called weighted visual-text cross alignment (WCA). This method begins with a localized visual prompting technique, designed to identify local visual areas within the query image. The local visual areas are then cross-aligned with the finer descriptions by creating a similarity matrix using the pre-trained VLM. To determine how well a query image aligns with each category, we develop a score function based on the weighted similarities in this matrix. Extensive experiments demonstrate that our method significantly improves zero-shot performance across various datasets, achieving results that are even comparable to few-shot learning methods.","url_abs":"https://arxiv.org/abs/2406.02915v1","url_pdf":"https://arxiv.org/pdf/2406.02915v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-text-cross-alignment-refining-the","repo_url":"https://github.com/tmlr-group/wca","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"semantic-image-text-similarity","task_name":"Semantic Image-Text Similarity"},{"task_slug":"visual-prompting","task_name":"Visual Prompting"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.02915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tmlr-group/WCA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tmlr-group/wca","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":8,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"tmlr-group/WCA","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"0614b1a2b2419fd7","entry":"construct_random","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0614b1a2b2419fd7"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"tmlr-group/WCA","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"dd3a895527885a09","entry":"load_classes","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3a895527885a09"}},{"code_sha256_prefix":"1069918b276d2855","entry":"load_json","repo":"tmlr-group/WCA","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/helper.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1069918b276d2855"}},{"code_sha256_prefix":"6775b1940ad09da2","entry":"make_descriptor_sentence","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6775b1940ad09da2"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"tmlr-group/WCA","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"1812ba359126089a","entry":"wordify","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1812ba359126089a"}},{"code_sha256_prefix":"39e6b23b55f376ea","entry":"build_model","repo":"tmlr-group/WCA","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39e6b23b55f376ea"}},{"code_sha256_prefix":"ed993071930bf283","entry":"generate_weights","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed993071930bf283"}},{"code_sha256_prefix":"2da7ec0975be872a","entry":"load","repo":"tmlr-group/WCA","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2da7ec0975be872a"}},{"code_sha256_prefix":"a989a043d984c92d","entry":"load_dataset","repo":"tmlr-group/WCA","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/WCA/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a989a043d984c92d"}},{"code_sha256_prefix":"9a15fb992127b666","entry":"zeroshot_classifier","repo":"tmlr-group/wca","repo_kind":"official","path":"helper.py","file_url":"https://github.com/tmlr-group/wca/blob/HEAD/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a15fb992127b666"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}