{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-12375","title":"Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning","arxiv_id":"2602.12375","date":"2026-02-12","proceeding":null,"authors":["Abdul Wahab","Raksha Kumaraswamy","Martha White"],"abstract":"Optimistic value estimates provide one mechanism for directed exploration in reinforcement learning (RL). The agent acts greedily with respect to an estimate of the value plus what can be seen as a value bonus. The value bonus can be learned by estimating a value function on reward bonuses, propagating local uncertainties around rewards. However, this approach only increases the value bonus for an action retroactively, after seeing a higher reward bonus from that state and action. Such an approach does not encourage the agent to visit a state and action for the first time. In this work, we introduce an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs). VBE uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration. The key idea is to design the rewards for these RQFs in such a way that the value bonus can decrease to zero. We show that VBE outperforms Bootstrap DQN and two reward bonus approaches (RND and ACB) on several classic environments used to test exploration and provide demonstrative experiments that it can scale easily to more complex environments like Atari.","url_abs":"https://arxiv.org/abs/2602.12375","url_pdf":"https://arxiv.org/pdf/2602.12375","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.12375","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.12375"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/mirzaabdulwahab1612/VBE","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"85924efd3389de23","entry":"LinearNet","repo":"mirzaabdulwahab1612/VBE","repo_kind":"found_in_text","path":"agent/exploratory_agents/OPI_outofsample_exploration.py","file_url":"https://github.com/mirzaabdulwahab1612/VBE/blob/HEAD/agent/exploratory_agents/OPI_outofsample_exploration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85924efd3389de23"}},{"code_sha256_prefix":"8ff3f8adb9181073","entry":"SimpleReplayBuffer","repo":"mirzaabdulwahab1612/VBE","repo_kind":"found_in_text","path":"agent/exploratory_agents/OPI_outofsample_exploration.py","file_url":"https://github.com/mirzaabdulwahab1612/VBE/blob/HEAD/agent/exploratory_agents/OPI_outofsample_exploration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ff3f8adb9181073"}},{"code_sha256_prefix":"a27f708e478bb398","entry":"SARSA","repo":"mirzaabdulwahab1612/VBE","repo_kind":"found_in_text","path":"agent/exploratory_agents/OPI_outofsample_exploration.py","file_url":"https://github.com/mirzaabdulwahab1612/VBE/blob/HEAD/agent/exploratory_agents/OPI_outofsample_exploration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a27f708e478bb398"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}