{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openxai-towards-a-transparent-evaluation-of","title":"OpenXAI: Towards a Transparent Evaluation of Model Explanations","arxiv_id":"2206.11104","date":"2022-06-22","proceeding":null,"authors":["Chirag Agarwal","Dan Ley","Satyapriya Krishna","Eshika Saxena","Martin Pawelczyk","Nari Johnson","Isha Puri","Marinka Zitnik","Himabindu Lakkaraju"],"abstract":"While several types of post hoc explanation methods have been proposed in recent literature, there is very little work on systematically benchmarking these methods. Here, we introduce OpenXAI, a comprehensive and extensible open-source framework for evaluating and benchmarking post hoc explanation methods. OpenXAI comprises of the following key components: (i) a flexible synthetic data generator and a collection of diverse real-world datasets, pre-trained models, and state-of-the-art feature attribution methods, and (ii) open-source implementations of eleven quantitative metrics for evaluating faithfulness, stability (robustness), and fairness of explanation methods, in turn providing comparisons of several explanation methods across a wide variety of metrics, models, and datasets. OpenXAI is easily extensible, as users can readily evaluate custom explanation methods and incorporate them into our leaderboards. Overall, OpenXAI provides an automated end-to-end pipeline that not only simplifies and standardizes the evaluation of post hoc explanation methods, but also promotes transparency and reproducibility in benchmarking these methods. While the first release of OpenXAI supports only tabular datasets, the explanation methods and metrics that we consider are general enough to be applicable to other data modalities. OpenXAI datasets and models, implementations of state-of-the-art explanation methods and evaluation metrics, are publicly available at this GitHub link.","url_abs":"https://arxiv.org/abs/2206.11104v5","url_pdf":"https://arxiv.org/pdf/2206.11104v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openxai-towards-a-transparent-evaluation-of","repo_url":"https://github.com/ai4life-group/openxai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"openxai-towards-a-transparent-evaluation-of","repo_url":"https://github.com/AI4LIFE-GROUP/LLM_Explainer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"xai","task_name":"Explainable Artificial Intelligence (XAI)"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[{"slug":"openxai","name":"OpenXAI","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.11104","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11104"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AI4LIFE-GROUP/LLM_Explainer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai4life-group/openxai","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6f7b5e5371682e42","entry":"LoadModel","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/model.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f7b5e5371682e42"}},{"code_sha256_prefix":"7000d93da4c642c2","entry":"ReturnLoaders","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/dataloader.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7000d93da4c642c2"}},{"code_sha256_prefix":"d475d4e1fbb7d7f5","entry":"ReturnTrainTestX","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/dataloader.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d475d4e1fbb7d7f5"}},{"code_sha256_prefix":"74ef36cde079b329","entry":"construct_param_string","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/experiment_utils.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/experiment_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74ef36cde079b329"}},{"code_sha256_prefix":"8f085348ccf769e4","entry":"convert_k_to_int","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/experiment_utils.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/experiment_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f085348ccf769e4"}},{"code_sha256_prefix":"08fd0c7cbb7cac6b","entry":"get_perturb_method","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/explainers/perturbation_methods.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/explainers/perturbation_methods.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08fd0c7cbb7cac6b"}},{"code_sha256_prefix":"9686c50c3c64a715","entry":"load_parameterized_file","repo":"ai4life-group/openxai","repo_kind":"official","path":"openxai/experiment_utils.py","file_url":"https://github.com/ai4life-group/openxai/blob/HEAD/openxai/experiment_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9686c50c3c64a715"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}