{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/calibration-of-p-values-for-calibration-and","title":"Calibration of P-values for calibration and for deviation of a subpopulation from the full population","arxiv_id":"2202.00100","date":"2022-01-31","proceeding":null,"authors":["Mark Tygert"],"abstract":"The author's recent research papers, \"Cumulative deviation of a subpopulation from the full population\" and \"A graphical method of cumulative differences between two subpopulations\" (both published in volume 8 of Springer's open-access \"Journal of Big Data\" during 2021), propose graphical methods and summary statistics, without extensively calibrating formal significance tests. The summary metrics and methods can measure the calibration of probabilistic predictions and can assess differences in responses between a subpopulation and the full population while controlling for a covariate or score via conditioning on it. These recently published papers construct significance tests based on the scalar summary statistics, but only sketch how to calibrate the attained significance levels (also known as \"P-values\") for the tests. The present article reviews and synthesizes work spanning many decades in order to detail how to calibrate the P-values. The present paper presents computationally efficient, easily implemented numerical methods for evaluating properly calibrated P-values, together with rigorous mathematical proofs guaranteeing their accuracy, and illustrates and validates the methods with open-source software and numerical examples.","url_abs":"https://arxiv.org/abs/2202.00100v7","url_pdf":"https://arxiv.org/pdf/2202.00100v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"calibration-of-p-values-for-calibration-and","repo_url":"https://github.com/facebookresearch/cdeets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"mathematical-proofs","task_name":"Mathematical Proofs"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.00100","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.00100"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/cdeets","reach":null}],"summary":{"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e6a46a9b7a1816d3","entry":"equierrs","repo":"facebookresearch/cdeets","repo_kind":"official","path":"codes/subpop_weighted.py","file_url":"https://github.com/facebookresearch/cdeets/blob/HEAD/codes/subpop_weighted.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6a46a9b7a1816d3"}},{"code_sha256_prefix":"aa7596acc59de1ee","entry":"cumulative","repo":"facebookresearch/cdeets","repo_kind":"official","path":"codes/subpop_weighted.py","file_url":"https://github.com/facebookresearch/cdeets/blob/HEAD/codes/subpop_weighted.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa7596acc59de1ee"}},{"code_sha256_prefix":"0f057f83ebd9e7dd","entry":"equiscores","repo":"facebookresearch/cdeets","repo_kind":"official","path":"codes/subpop_weighted.py","file_url":"https://github.com/facebookresearch/cdeets/blob/HEAD/codes/subpop_weighted.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f057f83ebd9e7dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}