{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bayesian-optimization-with-conformal-coverage","title":"Bayesian Optimization with Conformal Prediction Sets","arxiv_id":"2210.12496","date":"2022-10-22","proceeding":null,"authors":["Samuel Stanton","Wesley Maddox","Andrew Gordon Wilson"],"abstract":"Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects decisions (i.e. objective function queries) with maximal expected utility with respect to the posterior distribution of a Bayesian model, which quantifies reducible, epistemic uncertainty about query outcomes. In practice, subjectively implausible outcomes can occur regularly for two reasons: 1) model misspecification and 2) covariate shift. Conformal prediction is an uncertainty quantification method with coverage guarantees even for misspecified models and a simple mechanism to correct for covariate shift. We propose conformal Bayesian optimization, which directs queries towards regions of search space where the model predictions have guaranteed validity, and investigate its behavior on a suite of black-box optimization tasks and tabular ranking tasks. In many cases we find that query coverage can be significantly improved without harming sample-efficiency.","url_abs":"https://arxiv.org/abs/2210.12496v4","url_pdf":"https://arxiv.org/pdf/2210.12496v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bayesian-optimization-with-conformal-coverage","repo_url":"https://github.com/samuelstanton/conformal-bayesopt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"conformal-prediction","task_name":"Conformal Prediction"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"decision-making-under-uncertainty","task_name":"Decision Making Under Uncertainty"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"prediction-intervals","task_name":"Prediction Intervals"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.12496","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.12496"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/samuelstanton/conformal-bayesopt","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b080ad950cbb24b","entry":"baseline_candidate_split","repo":"samuelstanton/conformal-bayesopt","repo_kind":"official","path":"scripts/tab_bandits.py","file_url":"https://github.com/samuelstanton/conformal-bayesopt/blob/HEAD/scripts/tab_bandits.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b080ad950cbb24b"}},{"code_sha256_prefix":"be3754a85ebaa20c","entry":"get_src_df","repo":"samuelstanton/conformal-bayesopt","repo_kind":"official","path":"scripts/tab_bandits.py","file_url":"https://github.com/samuelstanton/conformal-bayesopt/blob/HEAD/scripts/tab_bandits.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be3754a85ebaa20c"}},{"code_sha256_prefix":"3cdb235923bd8c57","entry":"softplus","repo":"samuelstanton/conformal-bayesopt","repo_kind":"official","path":"conformalbo/acquisition/monte_carlo.py","file_url":"https://github.com/samuelstanton/conformal-bayesopt/blob/HEAD/conformalbo/acquisition/monte_carlo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cdb235923bd8c57"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}