{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rashomon-capacity-a-metric-for-predictive","title":"Rashomon Capacity: A Metric for Predictive Multiplicity in Classification","arxiv_id":"2206.01295","date":"2022-06-02","proceeding":null,"authors":["Hsiang Hsu","Flavio du Pin Calmon"],"abstract":"Predictive multiplicity occurs when classification models with statistically indistinguishable performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new metric, called Rashomon Capacity, to measure predictive multiplicity in probabilistic classification. Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes. In contrast, Rashomon Capacity applies to probabilistic classifiers, capturing more nuanced score variations for individual samples. We provide a rigorous derivation for Rashomon Capacity, argue its intuitive appeal, and demonstrate how to estimate it in practice. We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders. Our numerical experiments illustrate how Rashomon Capacity captures predictive multiplicity in various datasets and learning models, including neural networks. The tools introduced in this paper can help data scientists measure and report predictive multiplicity prior to model deployment.","url_abs":"https://arxiv.org/abs/2206.01295v2","url_pdf":"https://arxiv.org/pdf/2206.01295v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rashomon-capacity-a-metric-for-predictive","repo_url":"https://github.com/hsianghsu/rashomon-capacity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.01295","atlas_url":"https://app.syntology.ai/?focus=2206.01295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.01295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HsiangHsu/rashomon-capacity","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hsianghsu/rashomon-capacity","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a3325676492415f","entry":"blahut_arimoto","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"utils/capacity.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/utils/capacity.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a3325676492415f"}},{"code_sha256_prefix":"0d36fe6945bd1e95","entry":"evaluate","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"awp/utils/training.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/awp/utils/training.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d36fe6945bd1e95"}},{"code_sha256_prefix":"fdfaa5cc16be2ab7","entry":"evaluate_all_loss","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"awp/utils/training.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/awp/utils/training.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdfaa5cc16be2ab7"}},{"code_sha256_prefix":"5d320702f03bacd2","entry":"train_model","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"awp/utils/training.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/awp/utils/training.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d320702f03bacd2"}},{"code_sha256_prefix":"2444db312b373781","entry":"blahut_arimoto","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"sampling/utils/capacity.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/sampling/utils/capacity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2444db312b373781"}},{"code_sha256_prefix":"a786906d1b0a17c9","entry":"compute_capacity","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"sampling/utils/capacity.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/sampling/utils/capacity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a786906d1b0a17c9"}},{"code_sha256_prefix":"966f5c9ba151cdab","entry":"compute_capacity","repo":"hsianghsu/rashomon-capacity","repo_kind":"official","path":"utils/capacity.py","file_url":"https://github.com/hsianghsu/rashomon-capacity/blob/HEAD/utils/capacity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"966f5c9ba151cdab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}