{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-19956","title":"Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models","arxiv_id":"2605.19956","date":"2026-05-19","proceeding":"ICML","authors":["Jia-Wei Hai","Yijun Wang","Xiu-Shen Wei"],"abstract":"Vision-Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient attention rollout mechanism to identify semantically meaningful regions surviving under adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT .","url_abs":"https://arxiv.org/abs/2605.19956","url_pdf":"https://arxiv.org/pdf/2605.19956","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.19956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.19956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/SEU-VIPGroup/A-TPT","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"b0f936d4d6ae3b8c","entry":"accuracy","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"utils/tools.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0f936d4d6ae3b8c"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"clip/simple_tokenizer.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"1ca27622f8ca45e9","entry":"validate","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"utils/tools.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/utils/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ca27622f8ca45e9"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"clip/simple_tokenizer.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"9b45cb80d93fe980","entry":"wrapper_method","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"torchattacks/attack.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/torchattacks/attack.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b45cb80d93fe980"}},{"code_sha256_prefix":"ce030f573ae70489","entry":"build_model","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"clip/model.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce030f573ae70489"}},{"code_sha256_prefix":"c4f7c480f2fa7ba0","entry":"differential_evolution","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"torchattacks/attacks/_differential_evolution.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/torchattacks/attacks/_differential_evolution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4f7c480f2fa7ba0"}},{"code_sha256_prefix":"db594353b8cf5c9f","entry":"load","repo":"SEU-VIPGroup/A-TPT","repo_kind":"found_in_text","path":"clip/clip.py","file_url":"https://github.com/SEU-VIPGroup/A-TPT/blob/HEAD/clip/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db594353b8cf5c9f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}