{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-based-uncertainty-in-value-functions","title":"Model-Based Uncertainty in Value Functions","arxiv_id":"2302.12526","date":"2023-02-24","proceeding":null,"authors":["Carlos E. Luis","Alessandro G. Bottero","Julia Vinogradska","Felix Berkenkamp","Jan Peters"],"abstract":"We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation, but the over-approximation may result in inefficient exploration. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values and explicitly characterizes the gap in previous work. Moreover, our uncertainty quantification technique is easily integrated into common exploration strategies and scales naturally beyond the tabular setting by using standard deep reinforcement learning architectures. Experiments in difficult exploration tasks, both in tabular and continuous control settings, show that our sharper uncertainty estimates improve sample-efficiency.","url_abs":"https://arxiv.org/abs/2302.12526v2","url_pdf":"https://arxiv.org/pdf/2302.12526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-based-uncertainty-in-value-functions","repo_url":"https://github.com/boschresearch/ube-mbrl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"model-based-reinforcement-learning","task_name":"Model-based Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.12526","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.12526"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/boschresearch/ube-mbrl","reach":null}],"summary":{"ran_fixture":1,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4484d3a48f910ab9","entry":"compute_ube_var","repo":"boschresearch/ube-mbrl","repo_kind":"official","path":"ube_mbrl/tabular/toy_mdp_example.py","file_url":"https://github.com/boschresearch/ube-mbrl/blob/HEAD/ube_mbrl/tabular/toy_mdp_example.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4484d3a48f910ab9"}},{"code_sha256_prefix":"cf53ecbb5ae4594a","entry":"solve_bellman_equation","repo":"boschresearch/ube-mbrl","repo_kind":"official","path":"ube_mbrl/tabular/toy_mdp_example.py","file_url":"https://github.com/boschresearch/ube-mbrl/blob/HEAD/ube_mbrl/tabular/toy_mdp_example.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cf53ecbb5ae4594a"}},{"code_sha256_prefix":"1db379c85a4d5867","entry":"compute_pombu_rewards","repo":"boschresearch/ube-mbrl","repo_kind":"official","path":"ube_mbrl/tabular/toy_mdp_example.py","file_url":"https://github.com/boschresearch/ube-mbrl/blob/HEAD/ube_mbrl/tabular/toy_mdp_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1db379c85a4d5867"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}