{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adversarial-robustness-for-visual-grounding","title":"Adversarial Robustness for Visual Grounding of Multimodal Large Language Models","arxiv_id":"2405.09981","date":"2024-05-16","proceeding":null,"authors":["Kuofeng Gao","Yang Bai","Jiawang Bai","Yong Yang","Shu-Tao Xia"],"abstract":"Multi-modal Large Language Models (MLLMs) have recently achieved enhanced performance across various vision-language tasks including visual grounding capabilities. However, the adversarial robustness of visual grounding remains unexplored in MLLMs. To fill this gap, we use referring expression comprehension (REC) as an example task in visual grounding and propose three adversarial attack paradigms as follows. Firstly, untargeted adversarial attacks induce MLLMs to generate incorrect bounding boxes for each object. Besides, exclusive targeted adversarial attacks cause all generated outputs to the same target bounding box. In addition, permuted targeted adversarial attacks aim to permute all bounding boxes among different objects within a single image. Extensive experiments demonstrate that the proposed methods can successfully attack visual grounding capabilities of MLLMs. Our methods not only provide a new perspective for designing novel attacks but also serve as a strong baseline for improving the adversarial robustness for visual grounding of MLLMs.","url_abs":"https://arxiv.org/abs/2405.09981v1","url_pdf":"https://arxiv.org/pdf/2405.09981v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adversarial-robustness-for-visual-grounding","repo_url":"https://github.com/KuofengGao/MLLM-Grounding-Robustness","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.09981","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.09981"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KuofengGao/MLLM-Grounding-Robustness","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_draft_wrong":1,"ran":1,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"747d1d6a93d4f89d","entry":"list_of_str","repo":"KuofengGao/MLLM-Grounding-Robustness","repo_kind":"official","path":"eval_scripts/eval_ref.py","file_url":"https://github.com/KuofengGao/MLLM-Grounding-Robustness/blob/HEAD/eval_scripts/eval_ref.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"747d1d6a93d4f89d"}},{"code_sha256_prefix":"52eb6c41f97993a4","entry":"clamp","repo":"KuofengGao/MLLM-Grounding-Robustness","repo_kind":"official","path":"eval_scripts/target_to_all.py","file_url":"https://github.com/KuofengGao/MLLM-Grounding-Robustness/blob/HEAD/eval_scripts/target_to_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"52eb6c41f97993a4"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"KuofengGao/MLLM-Grounding-Robustness","repo_kind":"official","path":"minigpt4/models/base_model.py","file_url":"https://github.com/KuofengGao/MLLM-Grounding-Robustness/blob/HEAD/minigpt4/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}