{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-rashomon-importance-distribution-getting","title":"The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance","arxiv_id":"2309.13775","date":"2023-09-24","proceeding":"NeurIPS 2023 11","authors":["Jon Donnelly","Srikar Katta","Cynthia Rudin","Edward P. Browne"],"abstract":"Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available at https://github.com/jdonnelly36/Rashomon_Importance_Distribution.","url_abs":"https://arxiv.org/abs/2309.13775v4","url_pdf":"https://arxiv.org/pdf/2309.13775v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-rashomon-importance-distribution-getting","repo_url":"https://github.com/jdonnelly36/rashomon_importance_distribution","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.13775","atlas_url":"https://app.syntology.ai/?focus=2309.13775","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.13775"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jdonnelly36/Rashomon_Importance_Distribution","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jdonnelly36/rashomon_importance_distribution","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5acaf9665f78162d","entry":"hash_rows","repo":"jdonnelly36/Rashomon_Importance_Distribution","repo_kind":"official","path":"vi_utils.py","file_url":"https://github.com/jdonnelly36/Rashomon_Importance_Distribution/blob/HEAD/vi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5acaf9665f78162d"}},{"code_sha256_prefix":"5bb3103f2cacb7dc","entry":"perturb_divided","repo":"jdonnelly36/Rashomon_Importance_Distribution","repo_kind":"official","path":"vi_utils.py","file_url":"https://github.com/jdonnelly36/Rashomon_Importance_Distribution/blob/HEAD/vi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bb3103f2cacb7dc"}},{"code_sha256_prefix":"14cffe414ba5b280","entry":"trie_predict_recursive","repo":"jdonnelly36/Rashomon_Importance_Distribution","repo_kind":"official","path":"rashomon_importance_utils.py","file_url":"https://github.com/jdonnelly36/Rashomon_Importance_Distribution/blob/HEAD/rashomon_importance_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14cffe414ba5b280"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}