{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pfns-are-flexible-models-for-real-world","title":"PFNs4BO: In-Context Learning for Bayesian Optimization","arxiv_id":"2305.17535","date":"2023-05-27","proceeding":null,"authors":["Samuel Müller","Matthias Feurer","Noah Hollmann","Frank Hutter"],"abstract":"In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) through in-context learning on any prior distribution that can be efficiently sampled from. We describe how this flexibility can be exploited for surrogate modeling in BO. We use PFNs to mimic a naive Gaussian process (GP), an advanced GP, and a Bayesian Neural Network (BNN). In addition, we show how to incorporate further information into the prior, such as allowing hints about the position of optima (user priors), ignoring irrelevant dimensions, and performing non-myopic BO by learning the acquisition function. The flexibility underlying these extensions opens up vast possibilities for using PFNs for BO. We demonstrate the usefulness of PFNs for BO in a large-scale evaluation on artificial GP samples and three different hyperparameter optimization testbeds: HPO-B, Bayesmark, and PD1. We publish code alongside trained models at github.com/automl/PFNs4BO.","url_abs":"https://arxiv.org/abs/2305.17535v5","url_pdf":"https://arxiv.org/pdf/2305.17535v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pfns-are-flexible-models-for-real-world","repo_url":"https://github.com/automl/pfns4bo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"automl","task_name":"AutoML"},{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"gaussian-process","method_name":"Gaussian Process"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.17535","atlas_url":"https://app.syntology.ai/?focus=2305.17535","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.17535"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huawei-noah/HEBO","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/automl/PFNs4BO","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/uber/bayesmark","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/releaunifreiburg/HPO-B","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/automl/pfns4bo","reach":null}],"summary":{"ran":7,"ran_violates":1,"ran_draft_wrong":4,"unverified":11},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1},"found_in_text":{"samples":15,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"63119f7d9577e6aa","entry":"ExactGPLayer","repo":"releaunifreiburg/HPO-B","repo_kind":"found_in_text","path":"methods/fsbo/fsbo_modules.py","file_url":"https://github.com/releaunifreiburg/HPO-B/blob/HEAD/methods/fsbo/fsbo_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63119f7d9577e6aa"}},{"code_sha256_prefix":"a45fecfa89d66add","entry":"FSBO","repo":"releaunifreiburg/HPO-B","repo_kind":"found_in_text","path":"methods/fsbo/fsbo_modules.py","file_url":"https://github.com/releaunifreiburg/HPO-B/blob/HEAD/methods/fsbo/fsbo_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a45fecfa89d66add"}},{"code_sha256_prefix":"6b43f439029eab3f","entry":"MLP","repo":"releaunifreiburg/HPO-B","repo_kind":"found_in_text","path":"methods/fsbo/fsbo_modules.py","file_url":"https://github.com/releaunifreiburg/HPO-B/blob/HEAD/methods/fsbo/fsbo_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b43f439029eab3f"}},{"code_sha256_prefix":"eef091a071004fbd","entry":"Metric","repo":"releaunifreiburg/HPO-B","repo_kind":"found_in_text","path":"methods/fsbo/fsbo_modules.py","file_url":"https://github.com/releaunifreiburg/HPO-B/blob/HEAD/methods/fsbo/fsbo_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eef091a071004fbd"}},{"code_sha256_prefix":"75067bd5ded03a56","entry":"SeqBN","repo":"automl/pfns4bo","repo_kind":"official","path":"pfns4bo/transformer.py","file_url":"https://github.com/automl/pfns4bo/blob/HEAD/pfns4bo/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75067bd5ded03a56"}},{"code_sha256_prefix":"0fc24036211faaaa","entry":"TransformerEncoderDiffInit","repo":"automl/pfns4bo","repo_kind":"official","path":"pfns4bo/transformer.py","file_url":"https://github.com/automl/pfns4bo/blob/HEAD/pfns4bo/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0fc24036211faaaa"}},{"code_sha256_prefix":"a5e5843ed74ae0a8","entry":"TransformerEncoderLayer","repo":"automl/pfns4bo","repo_kind":"official","path":"pfns4bo/transformer.py","file_url":"https://github.com/automl/pfns4bo/blob/HEAD/pfns4bo/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5e5843ed74ae0a8"}},{"code_sha256_prefix":"5c9aeabdd9c0592a","entry":"bool_mask_to_att_mask","repo":"automl/pfns4bo","repo_kind":"official","path":"pfns4bo/transformer.py","file_url":"https://github.com/automl/pfns4bo/blob/HEAD/pfns4bo/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c9aeabdd9c0592a"}},{"code_sha256_prefix":"42aa387052bae4f8","entry":"log01","repo":"automl/PFNs4BO","repo_kind":"official","path":"pfns4bo/scripts/acquisition_functions.py","file_url":"https://github.com/automl/PFNs4BO/blob/HEAD/pfns4bo/scripts/acquisition_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42aa387052bae4f8"}},{"code_sha256_prefix":"863625149dc838e6","entry":"log01_batch","repo":"automl/PFNs4BO","repo_kind":"official","path":"pfns4bo/scripts/acquisition_functions.py","file_url":"https://github.com/automl/PFNs4BO/blob/HEAD/pfns4bo/scripts/acquisition_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"863625149dc838e6"}},{"code_sha256_prefix":"3d1bfce58b071127","entry":"lognormed_batch","repo":"automl/PFNs4BO","repo_kind":"official","path":"pfns4bo/scripts/acquisition_functions.py","file_url":"https://github.com/automl/PFNs4BO/blob/HEAD/pfns4bo/scripts/acquisition_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d1bfce58b071127"}},{"code_sha256_prefix":"a6b7d679f45f2735","entry":"totorch","repo":"releaunifreiburg/HPO-B","repo_kind":"found_in_text","path":"methods/fsbo/fsbo_modules.py","file_url":"https://github.com/releaunifreiburg/HPO-B/blob/HEAD/methods/fsbo/fsbo_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6b7d679f45f2735"}},{"code_sha256_prefix":"726be38598e80ba6","entry":"TransformerModel","repo":"automl/pfns4bo","repo_kind":"official","path":"pfns4bo/transformer.py","file_url":"https://github.com/automl/pfns4bo/blob/HEAD/pfns4bo/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"726be38598e80ba6"}},{"code_sha256_prefix":"d7f53b32976ced36","entry":"arg_safe_str","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/experiment_launcher.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/experiment_launcher.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d7f53b32976ced36"}},{"code_sha256_prefix":"e14ad665fb269166","entry":"arg_to_str","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/cmd_parse.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/cmd_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e14ad665fb269166"}},{"code_sha256_prefix":"ebc5e6b82ad05d37","entry":"expected_max","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/expected_max.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/expected_max.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebc5e6b82ad05d37"}},{"code_sha256_prefix":"67e25bdc1919213b","entry":"expected_min","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/expected_max.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/expected_max.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67e25bdc1919213b"}},{"code_sha256_prefix":"db39ace284e8c2b7","entry":"get_expected_max_weights","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/expected_max.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/expected_max.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db39ace284e8c2b7"}},{"code_sha256_prefix":"48c5d2f57a53830c","entry":"isclose_lte","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/np_util.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/np_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48c5d2f57a53830c"}},{"code_sha256_prefix":"00ee81526cac814f","entry":"namespace_to_dict","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/cmd_parse.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/cmd_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00ee81526cac814f"}},{"code_sha256_prefix":"aa2c8eadd32e1af0","entry":"random_seed","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/np_util.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/np_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa2c8eadd32e1af0"}},{"code_sha256_prefix":"70bd99eccb2d2020","entry":"serializable_dict","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/cmd_parse.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/cmd_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70bd99eccb2d2020"}},{"code_sha256_prefix":"a1efc44f1758e02c","entry":"strat_split","repo":"uber/bayesmark","repo_kind":"found_in_text","path":"bayesmark/np_util.py","file_url":"https://github.com/uber/bayesmark/blob/HEAD/bayesmark/np_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1efc44f1758e02c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}