{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2508-05430","title":"Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions","arxiv_id":"2508.05430","date":"2025-08-07","proceeding":"NeurIPS","authors":["Hubert Baniecki","Maximilian Muschalik","Fabian Fumagalli","Barbara Hammer","Eyke Hüllermeier","Przemyslaw Biecek"],"abstract":"Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the insertion/deletion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-1k benchmarks validate that second-order methods, such as FIxLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIxLIP in comparing different models, e.g. CLIP vs. SigLIP-2.","url_abs":"https://arxiv.org/abs/2508.05430","url_pdf":"https://arxiv.org/pdf/2508.05430","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2508.05430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2508.05430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/hbaniecki/fixlip","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a78226f57abd323","entry":"compute_weighted_r2","repo":"hbaniecki/fixlip","repo_kind":"found_in_text","path":"src/evaluation.py","file_url":"https://github.com/hbaniecki/fixlip/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a78226f57abd323"}},{"code_sha256_prefix":"0c31b6ee95169865","entry":"get_interesting_starting_players","repo":"hbaniecki/fixlip","repo_kind":"found_in_text","path":"src/clique.py","file_url":"https://github.com/hbaniecki/fixlip/blob/HEAD/src/clique.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c31b6ee95169865"}},{"code_sha256_prefix":"40939a7df8be6028","entry":"get_regression_weights","repo":"hbaniecki/fixlip","repo_kind":"found_in_text","path":"src/fixlip.py","file_url":"https://github.com/hbaniecki/fixlip/blob/HEAD/src/fixlip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40939a7df8be6028"}},{"code_sha256_prefix":"4e86fe764cb338b8","entry":"own_resize","repo":"hbaniecki/fixlip","repo_kind":"found_in_text","path":"src/plot.py","file_url":"https://github.com/hbaniecki/fixlip/blob/HEAD/src/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e86fe764cb338b8"}},{"code_sha256_prefix":"262c01e22127bd11","entry":"solve_regression","repo":"hbaniecki/fixlip","repo_kind":"found_in_text","path":"src/fixlip.py","file_url":"https://github.com/hbaniecki/fixlip/blob/HEAD/src/fixlip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"262c01e22127bd11"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}