{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-feature-attribution-an-information","title":"A Consistent and Efficient Evaluation Strategy for Attribution Methods","arxiv_id":"2202.00449","date":"2022-02-01","proceeding":null,"authors":["Yao Rong","Tobias Leemann","Vadim Borisov","Gjergji Kasneci","Enkelejda Kasneci"],"abstract":"With a variety of local feature attribution methods being proposed in recent years, follow-up work suggested several evaluation strategies. To assess the attribution quality across different attribution techniques, the most popular among these evaluation strategies in the image domain use pixel perturbations. However, recent advances discovered that different evaluation strategies produce conflicting rankings of attribution methods and can be prohibitively expensive to compute. In this work, we present an information-theoretic analysis of evaluation strategies based on pixel perturbations. Our findings reveal that the results are strongly affected by information leakage through the shape of the removed pixels as opposed to their actual values. Using our theoretical insights, we propose a novel evaluation framework termed Remove and Debias (ROAD) which offers two contributions: First, it mitigates the impact of the confounders, which entails higher consistency among evaluation strategies. Second, ROAD does not require the computationally expensive retraining step and saves up to 99% in computational costs compared to the state-of-the-art. We release our source code at https://github.com/tleemann/road_evaluation.","url_abs":"https://arxiv.org/abs/2202.00449v2","url_pdf":"https://arxiv.org/pdf/2202.00449v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-feature-attribution-an-information","repo_url":"https://github.com/tleemann/road_evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.00449","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.00449"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tleemann/road_evaluation","reach":null}],"summary":{"ran":8,"ran_honours":1,"ran_violates":1,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d39eb0a1bab6a0a","entry":"ChannelMeanImputer","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d39eb0a1bab6a0a"}},{"code_sha256_prefix":"a5a2de6a98b65ad1","entry":"ConvBlock_ResNet","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5a2de6a98b65ad1"}},{"code_sha256_prefix":"5e433e65a3f46c52","entry":"GNet_ResNet","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e433e65a3f46c52"}},{"code_sha256_prefix":"a75a12951c70b941","entry":"ImputedDataset","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a75a12951c70b941"}},{"code_sha256_prefix":"11860240064c7dd0","entry":"ImputedDatasetMasksOnly","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11860240064c7dd0"}},{"code_sha256_prefix":"47503ad265daf120","entry":"ImputingDataLoaderWrapper","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47503ad265daf120"}},{"code_sha256_prefix":"629765eba65036f5","entry":"NoisyLinearImputer","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"629765eba65036f5"}},{"code_sha256_prefix":"428d991a1dcea84e","entry":"ZeroImputer","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"428d991a1dcea84e"}},{"code_sha256_prefix":"b0613e306198627a","entry":"normalize_map","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0613e306198627a"}},{"code_sha256_prefix":"1d09d53a4f0a9512","entry":"rescale_channel","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d09d53a4f0a9512"}},{"code_sha256_prefix":"a9eaec23fad20830","entry":"BaseImputer","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9eaec23fad20830"}},{"code_sha256_prefix":"c676b8a8d65d31fe","entry":"GAINImputer","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c676b8a8d65d31fe"}},{"code_sha256_prefix":"b28b45f46cb768f9","entry":"SelfAttention","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b28b45f46cb768f9"}},{"code_sha256_prefix":"dd12bbba39b07ee5","entry":"_from_str","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd12bbba39b07ee5"}},{"code_sha256_prefix":"44de25a587d6939c","entry":"road_eval","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44de25a587d6939c"}},{"code_sha256_prefix":"5c9f538fc7ee4d08","entry":"run_road","repo":"tleemann/road_evaluation","repo_kind":"official","path":"road/road.py","file_url":"https://github.com/tleemann/road_evaluation/blob/HEAD/road/road.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c9f538fc7ee4d08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}