{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mining-fine-grained-image-text-alignment-for","title":"Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training","arxiv_id":"2401.02347","date":"2024-01-04","proceeding":null,"authors":["Longtian Qiu","Shan Ning","Xuming He"],"abstract":"Image captioning aims at generating descriptive and meaningful textual descriptions of images, enabling a broad range of vision-language applications. Prior works have demonstrated that harnessing the power of Contrastive Image Language Pre-training (CLIP) offers a promising approach to achieving zero-shot captioning, eliminating the need for expensive caption annotations. However, the widely observed modality gap in the latent space of CLIP harms the performance of zero-shot captioning by breaking the alignment between paired image-text features. To address this issue, we conduct an analysis on the CLIP latent space which leads to two findings. Firstly, we observe that the CLIP's visual feature of image subregions can achieve closer proximity to the paired caption due to the inherent information loss in text descriptions. In addition, we show that the modality gap between a paired image-text can be empirically modeled as a zero-mean Gaussian distribution. Motivated by the findings, we propose a novel zero-shot image captioning framework with text-only training to reduce the modality gap. In particular, we introduce a subregion feature aggregation to leverage local region information, which produces a compact visual representation for matching text representation. Moreover, we incorporate a noise injection and CLIP reranking strategy to boost captioning performance. We also extend our framework to build a zero-shot VQA pipeline, demonstrating its generality. Through extensive experiments on common captioning and VQA datasets such as MSCOCO, Flickr30k and VQAV2, we show that our method achieves remarkable performance improvements. Code is available at https://github.com/Artanic30/MacCap.","url_abs":"https://arxiv.org/abs/2401.02347v1","url_pdf":"https://arxiv.org/pdf/2401.02347v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mining-fine-grained-image-text-alignment-for","repo_url":"https://github.com/artanic30/maccap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"reranking","task_name":"Reranking"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.02347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.02347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Artanic30/MacCap","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"e5b8c6b0fc91e158","entry":"TransformerDecoder","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5b8c6b0fc91e158"}},{"code_sha256_prefix":"cd5d6fd1f5ffaa80","entry":"enlarge_past_key_values","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd5d6fd1f5ffaa80"}},{"code_sha256_prefix":"bac836298966f15b","entry":"select_past_key_values","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"bac836298966f15b"}},{"code_sha256_prefix":"58ca49dcd274ccf6","entry":"BaseModel","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58ca49dcd274ccf6"}},{"code_sha256_prefix":"72e4f7cf6aa18c2b","entry":"MacCap","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72e4f7cf6aa18c2b"}},{"code_sha256_prefix":"0778806c92cc4aba","entry":"PlugAndPlayContrastiveDecodingOneStepFast","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0778806c92cc4aba"}},{"code_sha256_prefix":"76ccbd16dce4027f","entry":"TransformerDecoderLayer","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76ccbd16dce4027f"}},{"code_sha256_prefix":"81e296736438ee23","entry":"plug_and_play_fast_ranking","repo":"Artanic30/MacCap","repo_kind":"official","path":"models/models/MacCap.py","file_url":"https://github.com/Artanic30/MacCap/blob/HEAD/models/models/MacCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81e296736438ee23"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}