{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-14837","title":"V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models","arxiv_id":"2509.14837","date":"2025-09-18","proceeding":"EMNLP","authors":["Qidong Wang","Junjie Hu","Ming Jiang"],"abstract":"Recent advances in causal interpretability have extended from language models to vision-language models (VLMs), seeking to reveal their internal mechanisms through input interventions. While textual interventions often target semantics, visual interventions typically rely on coarse pixel-level perturbations, limiting semantic insights on multimodal integration. In this study, we introduce V-SEAM, a novel framework that combines Visual Semantic Editing and Attention Modulating for causal interpretation of VLMs. V-SEAM enables concept-level visual manipulations and identifies attention heads with positive or negative contributions to predictions across three semantic levels: objects, attributes, and relationships. We observe that positive heads are often shared within the same semantic level but vary across levels, while negative heads tend to generalize broadly. Finally, we introduce an automatic method to modulate key head embeddings, demonstrating enhanced performance for both LLaVA and InstructBLIP across three diverse VQA benchmarks. Our data and code are released at: https://github.com/petergit1/V-SEAM.","url_abs":"https://arxiv.org/abs/2509.14837","url_pdf":"https://arxiv.org/pdf/2509.14837","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.14837","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.14837"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/petergit1/V-SEAM","reach":null}],"summary":{"ran":1,"ran_honours":1,"ran_fixture":2},"by_repo_kind":{"found_in_text":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"eaf8c1834b4b606d","entry":"_find_attn_module_for_layer","repo":"petergit1/V-SEAM","repo_kind":"found_in_text","path":"code/Attention-Modulate/Attention_Head_Rescaling.py","file_url":"https://github.com/petergit1/V-SEAM/blob/HEAD/code/Attention-Modulate/Attention_Head_Rescaling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eaf8c1834b4b606d"}},{"code_sha256_prefix":"7a13b05a78056d64","entry":"_get_num_heads","repo":"petergit1/V-SEAM","repo_kind":"found_in_text","path":"code/Attention-Modulate/Attention_Head_Rescaling.py","file_url":"https://github.com/petergit1/V-SEAM/blob/HEAD/code/Attention-Modulate/Attention_Head_Rescaling.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a13b05a78056d64"}},{"code_sha256_prefix":"8e417eb606ae2591","entry":"_make_layer_scaling_o_proj_pre_hook","repo":"petergit1/V-SEAM","repo_kind":"found_in_text","path":"code/Attention-Modulate/Attention_Head_Rescaling.py","file_url":"https://github.com/petergit1/V-SEAM/blob/HEAD/code/Attention-Modulate/Attention_Head_Rescaling.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e417eb606ae2591"}},{"code_sha256_prefix":"c57c7ca8f06c226d","entry":"register_head_edit_hooks","repo":"petergit1/V-SEAM","repo_kind":"found_in_text","path":"code/Attention-Modulate/Attention_Head_Rescaling.py","file_url":"https://github.com/petergit1/V-SEAM/blob/HEAD/code/Attention-Modulate/Attention_Head_Rescaling.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c57c7ca8f06c226d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}