{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/attention-prompting-on-image-for-large-vision","title":"Attention Prompting on Image for Large Vision-Language Models","arxiv_id":"2409.17143","date":"2024-09-25","proceeding":null,"authors":["Runpeng Yu","Weihao Yu","Xinchao Wang"],"abstract":"Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.","url_abs":"https://arxiv.org/abs/2409.17143v1","url_pdf":"https://arxiv.org/pdf/2409.17143v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"attention-prompting-on-image-for-large-vision","repo_url":"https://github.com/yu-rp/apiprompting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":null,"task_name":"MM-Vet"},{"task_slug":"visual-prompting","task_name":"Visual Prompting"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"heatmap","method_name":"Heatmap"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.17143","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.17143"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yu-rp/apiprompting","reach":null}],"summary":{"ran":3,"ran_draft_wrong":3,"ran_honours":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"8992e3094a2ca1d1","entry":"Conversation","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8992e3094a2ca1d1"}},{"code_sha256_prefix":"2fe03dcbac95b31f","entry":"KeywordsStoppingCriteria","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fe03dcbac95b31f"}},{"code_sha256_prefix":"e1119b92480f0cdc","entry":"MaxNewTokensCriteria","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1119b92480f0cdc"}},{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"a7bee88c1c7fd6a3","entry":"image_parser","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a7bee88c1c7fd6a3"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"d5044bdde33c54c2","entry":"load_images","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5044bdde33c54c2"}},{"code_sha256_prefix":"e8f3a2a1be36a4d7","entry":"process_images","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8f3a2a1be36a4d7"}},{"code_sha256_prefix":"0c4ff52904c9e0e7","entry":"SeparatorStyle","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c4ff52904c9e0e7"}},{"code_sha256_prefix":"724da83b5bf10219","entry":"StoppingCriteria","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"724da83b5bf10219"}},{"code_sha256_prefix":"62a88066e939d2c5","entry":"disable_torch_init","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62a88066e939d2c5"}},{"code_sha256_prefix":"6565708bd99b9f3b","entry":"getmask","repo":"yu-rp/apiprompting","repo_kind":"official","path":"API/API_LLaVA/functions.py","file_url":"https://github.com/yu-rp/apiprompting/blob/HEAD/API/API_LLaVA/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6565708bd99b9f3b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}