{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-the-attribution-quality-of","title":"Benchmarking the Attribution Quality of Vision Models","arxiv_id":"2407.11910","date":"2024-07-16","proceeding":null,"authors":["Robin Hesse","Simone Schaub-Meyer","Stefan Roth"],"abstract":"Attribution maps are one of the most established tools to explain the functioning of computer vision models. They assign importance scores to input features, indicating how relevant each feature is for the prediction of a deep neural network. While much research has gone into proposing new attribution methods, their proper evaluation remains a difficult challenge. In this work, we propose a novel evaluation protocol that overcomes two fundamental limitations of the widely used incremental-deletion protocol, i.e., the out-of-domain issue and lacking inter-model comparisons. This allows us to evaluate 23 attribution methods and how different design choices of popular vision backbones affect their attribution quality. We find that intrinsically explainable models outperform standard models and that raw attribution values exhibit a higher attribution quality than what is known from previous work. Further, we show consistent changes in the attribution quality when varying the network design, indicating that some standard design choices promote attribution quality.","url_abs":"https://arxiv.org/abs/2407.11910v2","url_pdf":"https://arxiv.org/pdf/2407.11910v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-the-attribution-quality-of","repo_url":"https://github.com/visinf/idsds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"explainable-models","task_name":"Explainable Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.11910","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.11910"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/visinf/idsds","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":2,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":14,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"927e684e71ca3ee9","entry":"compute_rollout_attention","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_LRP.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"927e684e71ca3ee9"}},{"code_sha256_prefix":"5a98eb7226d25564","entry":"compute_rollout_attention","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_explanation_generator.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_explanation_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a98eb7226d25564"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"visinf/idsds","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"visinf/idsds","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"5adf613fdebfcbd4","entry":"make_layers","repo":"visinf/idsds","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5adf613fdebfcbd4"}},{"code_sha256_prefix":"8c0113732a399cea","entry":"resnet18","repo":"visinf/idsds","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c0113732a399cea"}},{"code_sha256_prefix":"76b1bf0c7e996f90","entry":"vgg11","repo":"visinf/idsds","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76b1bf0c7e996f90"}},{"code_sha256_prefix":"df6cfaa1fbccd91a","entry":"vgg11_bn","repo":"visinf/idsds","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df6cfaa1fbccd91a"}},{"code_sha256_prefix":"5fe2124317452946","entry":"vit_base_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_LRP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fe2124317452946"}},{"code_sha256_prefix":"5406cd8a829fda92","entry":"vit_base_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_gradient_attribution.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_gradient_attribution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5406cd8a829fda92"}},{"code_sha256_prefix":"57e03ac784886a52","entry":"vit_base_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_new.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_new.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57e03ac784886a52"}},{"code_sha256_prefix":"005867bd71baa4bb","entry":"vit_large_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_LRP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"005867bd71baa4bb"}},{"code_sha256_prefix":"1b096e0f5504ad57","entry":"vit_large_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_gradient_attribution.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_gradient_attribution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b096e0f5504ad57"}},{"code_sha256_prefix":"fd65fde98f9bb5fc","entry":"vit_large_patch16_224","repo":"visinf/idsds","repo_kind":"official","path":"models/ViT/ViT_new.py","file_url":"https://github.com/visinf/idsds/blob/HEAD/models/ViT/ViT_new.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd65fde98f9bb5fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}