{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-song-of-dis-agreement-evaluating-the","title":"A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing","arxiv_id":"2205.04559","date":"2022-05-09","proceeding":null,"authors":["Michael Neely","Stefan F. Schouten","Maurits Bleeker","Ana Lucic"],"abstract":"There has been significant debate in the NLP community about whether or not attention weights can be used as an explanation - a mechanism for interpreting how important each input token is for a particular prediction. The validity of \"attention as explanation\" has so far been evaluated by computing the rank correlation between attention-based explanations and existing feature attribution explanations using LSTM-based models. In our work, we (i) compare the rank correlation between five more recent feature attribution methods and two attention-based methods, on two types of NLP tasks, and (ii) extend this analysis to also include transformer-based models. We find that attention-based explanations do not correlate strongly with any recent feature attribution methods, regardless of the model or task. Furthermore, we find that none of the tested explanations correlate strongly with one another for the transformer-based model, leading us to question the underlying assumption that we should measure the validity of attention-based explanations based on how well they correlate with existing feature attribution explanation methods. After conducting experiments on five datasets using two different models, we argue that the community should stop using rank correlation as an evaluation metric for attention-based explanations. We suggest that researchers and practitioners should instead test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand.","url_abs":"https://arxiv.org/abs/2205.04559v1","url_pdf":"https://arxiv.org/pdf/2205.04559v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-song-of-dis-agreement-evaluating-the","repo_url":"https://github.com/sfschouten/court-of-xai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"explainable-artificial-intelligence","task_name":"Explainable artificial intelligence"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.04559","atlas_url":"https://app.syntology.ai/?focus=2205.04559","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.04559"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sfschouten/court-of-xai","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ec174e94712e2180","entry":"bucket_order","repo":"sfschouten/court-of-xai","repo_kind":"official","path":"xai_court/common/kendall_top_k.py","file_url":"https://github.com/sfschouten/court-of-xai/blob/HEAD/xai_court/common/kendall_top_k.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec174e94712e2180"}},{"code_sha256_prefix":"9a6ae2c6a51f3295","entry":"ensure_dir","repo":"sfschouten/court-of-xai","repo_kind":"official","path":"xai_court/common/utils.py","file_url":"https://github.com/sfschouten/court-of-xai/blob/HEAD/xai_court/common/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a6ae2c6a51f3295"}},{"code_sha256_prefix":"5dadd6853ceb3f28","entry":"kendall_top_k","repo":"sfschouten/court-of-xai","repo_kind":"official","path":"xai_court/common/kendall_top_k.py","file_url":"https://github.com/sfschouten/court-of-xai/blob/HEAD/xai_court/common/kendall_top_k.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dadd6853ceb3f28"}},{"code_sha256_prefix":"ee1d4ed65f18adf6","entry":"model_already_trained","repo":"sfschouten/court-of-xai","repo_kind":"official","path":"xai_court/common/utils.py","file_url":"https://github.com/sfschouten/court-of-xai/blob/HEAD/xai_court/common/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee1d4ed65f18adf6"}},{"code_sha256_prefix":"79915b1ece5d2157","entry":"enforce_same_shape","repo":"sfschouten/court-of-xai","repo_kind":"official","path":"xai_court/common/correlation_measures.py","file_url":"https://github.com/sfschouten/court-of-xai/blob/HEAD/xai_court/common/correlation_measures.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79915b1ece5d2157"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}