{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-diagnostic-study-of-explainability","title":"A Diagnostic Study of Explainability Techniques for Text Classification","arxiv_id":"2009.13295","date":"2020-09-25","proceeding":"EMNLP 2020 11","authors":["Pepa Atanasova","Jakob Grue Simonsen","Christina Lioma","Isabelle Augenstein"],"abstract":"Recent developments in machine learning have introduced models that approach human performance at the cost of increased architectural complexity. Efforts to make the rationales behind the models' predictions transparent have inspired an abundance of new explainability techniques. Provided with an already trained model, they compute saliency scores for the words of an input instance. However, there exists no definitive guide on (i) how to choose such a technique given a particular application task and model architecture, and (ii) the benefits and drawbacks of using each such technique. In this paper, we develop a comprehensive list of diagnostic properties for evaluating existing explainability techniques. We then employ the proposed list to compare a set of diverse explainability techniques on downstream text classification tasks and neural network architectures. We also compare the saliency scores assigned by the explainability techniques with human annotations of salient input regions to find relations between a model's performance and the agreement of its rationales with human ones. Overall, we find that the gradient-based explanations perform best across tasks and model architectures, and we present further insights into the properties of the reviewed explainability techniques.","url_abs":"https://arxiv.org/abs/2009.13295v1","url_pdf":"https://arxiv.org/pdf/2009.13295v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-diagnostic-study-of-explainability","repo_url":"https://github.com/copenlu/xai-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.13295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.13295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/copenlu/xai-benchmark","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e060ae69644f44d3","entry":"get_layer_activation","repo":"copenlu/xai-benchmark","repo_kind":"official","path":"saliency_eval/consistency_rats.py","file_url":"https://github.com/copenlu/xai-benchmark/blob/HEAD/saliency_eval/consistency_rats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e060ae69644f44d3"}},{"code_sha256_prefix":"b32a74f1e30ad9f2","entry":"get_model_dist","repo":"copenlu/xai-benchmark","repo_kind":"official","path":"saliency_eval/consistency_rats.py","file_url":"https://github.com/copenlu/xai-benchmark/blob/HEAD/saliency_eval/consistency_rats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b32a74f1e30ad9f2"}},{"code_sha256_prefix":"549c73a3ed87c729","entry":"save_activation","repo":"copenlu/xai-benchmark","repo_kind":"official","path":"saliency_eval/consistency_rats.py","file_url":"https://github.com/copenlu/xai-benchmark/blob/HEAD/saliency_eval/consistency_rats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"549c73a3ed87c729"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}