{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learn-then-test-calibrating-predictive","title":"Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control","arxiv_id":"2110.01052","date":"2021-10-03","proceeding":null,"authors":["Anastasios N. Angelopoulos","Stephen Bates","Emmanuel J. Candès","Michael I. Jordan","Lihua Lei"],"abstract":"We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.","url_abs":"https://arxiv.org/abs/2110.01052v5","url_pdf":"https://arxiv.org/pdf/2110.01052v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learn-then-test-calibrating-predictive","repo_url":"https://github.com/aangelopoulos/ltt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learn-then-test-calibrating-predictive","repo_url":"https://github.com/kclip/r_autoeval_plus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"outlier-detection","task_name":"Outlier Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.01052","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.01052"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kclip/r_autoeval_plus","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aangelopoulos/ltt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":3,"ran_violates":3,"unverified":16},"by_repo_kind":{"official":{"samples":22,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b8d750f24ed95e2","entry":"expand_grid","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/uniform_concentration.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/uniform_concentration.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b8d750f24ed95e2"}},{"code_sha256_prefix":"86294eacaecc7ea5","entry":"h1","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/bounds.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/bounds.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86294eacaecc7ea5"}},{"code_sha256_prefix":"cbecdda63cea8201","entry":"h2","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/bounds.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/bounds.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbecdda63cea8201"}},{"code_sha256_prefix":"47b7bbfae0285051","entry":"hoeffding_plus","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/bounds.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/bounds.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47b7bbfae0285051"}},{"code_sha256_prefix":"8f4174eb8bea74e1","entry":"safe_min","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/uniform_concentration.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/uniform_concentration.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f4174eb8bea74e1"}},{"code_sha256_prefix":"ce9cdd0a73c5e072","entry":"sort_sum","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/utils.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce9cdd0a73c5e072"}},{"code_sha256_prefix":"4011969a3bfc90e8","entry":"cacheable","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/uniform_concentration.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/uniform_concentration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4011969a3bfc90e8"}},{"code_sha256_prefix":"0587a692aa8a759d","entry":"cacheable","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0587a692aa8a759d"}},{"code_sha256_prefix":"da5d46b4724c02dd","entry":"eval_detector","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/detection/loss_table.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/detection/loss_table.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da5d46b4724c02dd"}},{"code_sha256_prefix":"9a6b6722d0017533","entry":"eval_image","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/detection/loss_table.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/detection/loss_table.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a6b6722d0017533"}},{"code_sha256_prefix":"7ee13d7a01cb35a2","entry":"formatFloat","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/table.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/table.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ee13d7a01cb35a2"}},{"code_sha256_prefix":"91a92d35cc14ae7d","entry":"get_images","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/grid_fig.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/grid_fig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91a92d35cc14ae7d"}},{"code_sha256_prefix":"6804c5f0e3efce41","entry":"get_lambdas_vs_pfdps_frac_predict","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/lambda_vs_pfdr.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/lambda_vs_pfdr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6804c5f0e3efce41"}},{"code_sha256_prefix":"77bf34d6bcd8a4dd","entry":"get_nus_rs_n","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/pfdr.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/pfdr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77bf34d6bcd8a4dd"}},{"code_sha256_prefix":"71af0e9d6912d492","entry":"pfdr_loss_table","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/pfdr.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/pfdr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71af0e9d6912d492"}},{"code_sha256_prefix":"c23944476f3cabe3","entry":"platt_logits","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/grid_fig.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/grid_fig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c23944476f3cabe3"}},{"code_sha256_prefix":"3f8f32096b61daee","entry":"quantile","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/table.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/table.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f8f32096b61daee"}},{"code_sha256_prefix":"dbcdd2b46781b966","entry":"risk_size","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/utils.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbcdd2b46781b966"}},{"code_sha256_prefix":"6fc5e8b6054560fc","entry":"romano_wolf","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/concentration.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/concentration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fc5e8b6054560fc"}},{"code_sha256_prefix":"394b954980f78fc7","entry":"romano_wolf_CLT","repo":"aangelopoulos/ltt","repo_kind":"official","path":"core/concentration.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/core/concentration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"394b954980f78fc7"}},{"code_sha256_prefix":"4d529ca40b514587","entry":"trial_precomputed","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/pfdr_histogram.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/pfdr_histogram.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d529ca40b514587"}},{"code_sha256_prefix":"bc281c57b11f153e","entry":"validate","repo":"aangelopoulos/ltt","repo_kind":"official","path":"experiments/imagenet/utils.py","file_url":"https://github.com/aangelopoulos/ltt/blob/HEAD/experiments/imagenet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc281c57b11f153e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}