{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-attention-model-explainability","title":"Rethinking Attention-Model Explainability through Faithfulness Violation Test","arxiv_id":"2201.12114","date":"2022-01-28","proceeding":null,"authors":["Yibing Liu","Haoliang Li","Yangyang Guo","Chenqi Kong","Jing Li","Shiqi Wang"],"abstract":"Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.","url_abs":"https://arxiv.org/abs/2201.12114v3","url_pdf":"https://arxiv.org/pdf/2201.12114v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-attention-model-explainability","repo_url":"https://github.com/BierOne/Attention-Faithfulness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"feature-importance","task_name":"Feature Importance"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2201.12114","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.12114"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BierOne/Attention-Faithfulness","reach":null}],"summary":{"ran_violates":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff598e6f64baea12","entry":"calculate_violators","repo":"BierOne/Attention-Faithfulness","repo_kind":"official","path":"Transparency/common_code/metrics.py","file_url":"https://github.com/BierOne/Attention-Faithfulness/blob/HEAD/Transparency/common_code/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff598e6f64baea12"}},{"code_sha256_prefix":"3211e78542fdee75","entry":"compute_rollout_attention","repo":"bierone/attention-faithfulness","repo_kind":"official","path":"Transformer-MM-Explainability/VisualBERT/mmf/models/transformers/backends/BERT_ours.py","file_url":"https://github.com/bierone/attention-faithfulness/blob/HEAD/Transformer-MM-Explainability/VisualBERT/mmf/models/transformers/backends/BERT_ours.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3211e78542fdee75"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}