{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2511-18891","title":"Reproducibility Study of Large Language Model Bayesian Optimization","arxiv_id":"2511.18891","date":"2025-11-24","proceeding":"ICLR","authors":["Adam Rychert","Gasper Spagnolo","Evgenii Posashkov"],"abstract":"In this reproducibility study, we revisit the LLAMBO framework of Daxberger et al. (2024), a prompting-based Bayesian optimization (BO) method that uses large language models as discriminative surrogates and acquisition optimizers via text-only interactions. We replicate the core Bayesmark and HPOBench experiments under the original evaluation protocol, but replace GPT-3.5 with the open-weight Llama 3.1 70B model used for all text encoding components. Our results broadly confirm the main claims of LLAMBO. Contextual warm starting via textual problem and hyperparameter descriptions substantially improves early regret behaviour and reduces variance across runs. LLAMBO's discriminative surrogate is weaker than GP or SMAC as a pure single task regressor, yet benefits from cross task semantic priors induced by the language model. Ablations that remove textual context markedly degrade predictive accuracy and calibration, while the LLAMBO candidate sampler consistently generates higher quality and more diverse proposals than TPE or random sampling. Experiments with smaller backbones (Gemma 27B, Llama 3.1 8B) yield unstable or invalid predictions, suggesting insufficient capacity for reliable surrogate behaviour. Overall, our study shows that the LLAMBO architecture is robust to changing the language model backbone and remains effective when instantiated with Llama 3.1 70B.","url_abs":"https://arxiv.org/abs/2511.18891","url_pdf":"https://arxiv.org/pdf/2511.18891","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2511.18891","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2511.18891"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/spagnoloG/llambo-reproducibility","reach":{"status":"ok"}}],"summary":{"unverified":19},"by_repo_kind":{"found_in_text":{"samples":19,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"cba63f2439729045","entry":"build_task_context","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure5_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure5_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cba63f2439729045"}},{"code_sha256_prefix":"ea2ad7622a230331","entry":"classify_points","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure5_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure5_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea2ad7622a230331"}},{"code_sha256_prefix":"f3706b1f61f7411c","entry":"compute_curve","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure6_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure6_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3706b1f61f7411c"}},{"code_sha256_prefix":"01dcb2bb80685795","entry":"compute_regrets","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure2_full.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure2_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01dcb2bb80685795"}},{"code_sha256_prefix":"821aaa6aa28279ce","entry":"compute_simple_regret","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure2_simple.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure2_simple.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"821aaa6aa28279ce"}},{"code_sha256_prefix":"254a84d6fe230b8e","entry":"compute_statistics","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/compare_with_baselines.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/compare_with_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"254a84d6fe230b8e"}},{"code_sha256_prefix":"71950af7498f601f","entry":"detect_models","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure3_surrogate.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure3_surrogate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71950af7498f601f"}},{"code_sha256_prefix":"4eaa242e74d66ce5","entry":"detect_models","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure4_ablation.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure4_ablation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4eaa242e74d66ce5"}},{"code_sha256_prefix":"68f5fcace926942e","entry":"get_bayesmark_func","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/bayesmark/bbox_utils.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/bayesmark/bbox_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68f5fcace926942e"}},{"code_sha256_prefix":"de3c790a760c352b","entry":"load_ablation_results","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure4_ablation.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure4_ablation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de3c790a760c352b"}},{"code_sha256_prefix":"0c6b34433ccdefc5","entry":"load_baseline_results","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/compare_with_baselines.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/compare_with_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c6b34433ccdefc5"}},{"code_sha256_prefix":"12ec5e69fc6f6a0f","entry":"load_dataset","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure5_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure5_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12ec5e69fc6f6a0f"}},{"code_sha256_prefix":"242b607add7e2954","entry":"load_init_dataframe","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure2_full.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure2_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"242b607add7e2954"}},{"code_sha256_prefix":"95733e7e8da65f81","entry":"load_llambo_results","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/compare_with_baselines.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/compare_with_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95733e7e8da65f81"}},{"code_sha256_prefix":"df7f0baf92afe57b","entry":"load_method_data","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure2_simple.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure2_simple.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df7f0baf92afe57b"}},{"code_sha256_prefix":"fb5374b98bbdc0da","entry":"load_metric_values","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure6_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure6_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb5374b98bbdc0da"}},{"code_sha256_prefix":"ce7b65c82f4f272d","entry":"load_metrics","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure2_full.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure2_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce7b65c82f4f272d"}},{"code_sha256_prefix":"5dd509e1de869639","entry":"load_results","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure3_surrogate.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure3_surrogate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5dd509e1de869639"}},{"code_sha256_prefix":"a264ff14fc9f320c","entry":"prepare_storage","repo":"spagnoloG/llambo-reproducibility","repo_kind":"found_in_text","path":"LLAMBO/plot_figure6_sampling.py","file_url":"https://github.com/spagnoloG/llambo-reproducibility/blob/HEAD/LLAMBO/plot_figure6_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a264ff14fc9f320c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}