{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bots-batch-bayesian-optimization-of-extended","title":"BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settings","arxiv_id":"2412.00308","date":"2024-11-30","proceeding":null,"authors":["Karine Karine","Susan A. Murphy","Benjamin M. Marlin"],"abstract":"In settings where the application of reinforcement learning (RL) requires running real-world trials, including the optimization of adaptive health interventions, the number of episodes available for learning can be severely limited due to cost or time constraints. In this setting, the bias-variance trade-off of contextual bandit methods can be significantly better than that of more complex full RL methods. However, Thompson sampling bandits are limited to selecting actions based on distributions of immediate rewards. In this paper, we extend the linear Thompson sampling bandit to select actions based on a state-action utility function consisting of the Thompson sampler's estimate of the expected immediate reward combined with an action bias term. We use batch Bayesian optimization over episodes to learn the action bias terms with the goal of maximizing the expected return of the extended Thompson sampler. The proposed approach is able to learn optimal policies for a strictly broader class of Markov decision processes (MDPs) than standard Thompson sampling. Using an adaptive intervention simulation environment that captures key aspects of behavioral dynamics, we show that the proposed method can significantly out-perform standard Thompson sampling in terms of total return, while requiring significantly fewer episodes than standard value function and policy gradient methods.","url_abs":"https://arxiv.org/abs/2412.00308v1","url_pdf":"https://arxiv.org/pdf/2412.00308v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"thompson-sampling","task_name":"Thompson Sampling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.00308","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.00308"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/reml-lab/BOTS","reach":null}],"summary":{"ran":2,"ran_fixture":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"fd3dedd297a20368","entry":"Packing","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd3dedd297a20368"}},{"code_sha256_prefix":"625214e9a51d1e55","entry":"TurboState","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"625214e9a51d1e55"}},{"code_sha256_prefix":"28d83db80a6ea5f6","entry":"update_TurboState","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28d83db80a6ea5f6"}},{"code_sha256_prefix":"8b154d91799aef45","entry":"BatchAgentBO","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b154d91799aef45"}},{"code_sha256_prefix":"c5ee07841a004ecc","entry":"BatchAgentBase","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5ee07841a004ecc"}},{"code_sha256_prefix":"a22ff25e1228ec7d","entry":"ThompsonSamplingBayesLRAgent","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a22ff25e1228ec7d"}},{"code_sha256_prefix":"c46051f5c5414c1d","entry":"create_TS_and_set_priors","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c46051f5c5414c1d"}},{"code_sha256_prefix":"43b92bdeec36b7ad","entry":"generate_TuRBO_batch","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43b92bdeec36b7ad"}},{"code_sha256_prefix":"d7493c4b601a1c89","entry":"get_TS_coeffs","repo":"reml-lab/BOTS","repo_kind":"found_in_text","path":"BOTS/batch_agents.py","file_url":"https://github.com/reml-lab/BOTS/blob/HEAD/BOTS/batch_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7493c4b601a1c89"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}