{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transferable-decoding-with-visual-entities","title":"Transferable Decoding with Visual Entities for Zero-Shot Image Captioning","arxiv_id":"2307.16525","date":"2023-07-31","proceeding":"ICCV 2023 1","authors":["Junjie Fei","Teng Wang","Jinrui Zhang","Zhenyu He","Chengjie Wang","Feng Zheng"],"abstract":"Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progress. However, we have observed and empirically demonstrated that these methods are susceptible to modality bias induced by LLMs and tend to generate descriptions containing objects (entities) that do not actually exist in the image but frequently appear during training (i.e., object hallucination). In this paper, we propose ViECap, a transferable decoding model that leverages entity-aware decoding to generate descriptions in both seen and unseen scenarios. ViECap incorporates entity-aware hard prompts to guide LLMs' attention toward the visual entities present in the image, enabling coherent caption generation across diverse scenes. With entity-aware hard prompts, ViECap is capable of maintaining performance when transferring from in-domain to out-of-domain scenarios. Extensive experiments demonstrate that ViECap sets a new state-of-the-art cross-domain (transferable) captioning and performs competitively in-domain captioning compared to previous VLMs-based zero-shot methods. Our code is available at: https://github.com/FeiElysia/ViECap","url_abs":"https://arxiv.org/abs/2307.16525v1","url_pdf":"https://arxiv.org/pdf/2307.16525v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transferable-decoding-with-visual-entities","repo_url":"https://github.com/feielysia/viecap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"object-hallucination","task_name":"Object Hallucination"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.16525","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.16525"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/CLIP","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/feielysia/viecap","reach":null}],"summary":{"ran":5,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"c28ba0c055107433","entry":"MappingNetwork","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c28ba0c055107433"}},{"code_sha256_prefix":"f324b01c64c51914","entry":"MlpTransformer","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f324b01c64c51914"}},{"code_sha256_prefix":"d8ba41633b35c84c","entry":"MultiHeadAttention","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8ba41633b35c84c"}},{"code_sha256_prefix":"07289a2cf68fd893","entry":"Transformer","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07289a2cf68fd893"}},{"code_sha256_prefix":"6436669ce470a3fd","entry":"TransformerLayer","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6436669ce470a3fd"}},{"code_sha256_prefix":"76aed27dc8d4286d","entry":"get_language_mode","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76aed27dc8d4286d"}},{"code_sha256_prefix":"9827fa8337846b02","entry":"ClipCaptionModel","repo":"feielysia/viecap","repo_kind":"official","path":"ClipCap.py","file_url":"https://github.com/feielysia/viecap/blob/HEAD/ClipCap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9827fa8337846b02"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}