{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reward-uncertainty-for-exploration-in-1","title":"Reward Uncertainty for Exploration in Preference-based Reinforcement Learning","arxiv_id":"2205.12401","date":"2022-05-24","proceeding":"ICLR 2022 4","authors":["Xinran Liang","Katherine Shu","Kimin Lee","Pieter Abbeel"],"abstract":"Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, recent exploration methods have proven to be a recipe for improving sample-efficiency in RL. We present an exploration method specifically for preference-based RL algorithms. Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward. Specifically, we utilize disagreement across ensemble of learned reward models. Our intuition is that disagreement in learned reward model reflects uncertainty in tailored human feedback and could be useful for exploration. Our experiments show that exploration bonus from uncertainty in learned reward improves both feedback- and sample-efficiency of preference-based RL algorithms on complex robot manipulation tasks from MetaWorld benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.","url_abs":"https://arxiv.org/abs/2205.12401v1","url_pdf":"https://arxiv.org/pdf/2205.12401v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reward-uncertainty-for-exploration-in-1","repo_url":"https://github.com/apple/ml-reed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reward-uncertainty-for-exploration-in-1","repo_url":"https://github.com/rll-research/rune","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.12401","atlas_url":"https://app.syntology.ai/?focus=2205.12401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.12401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rll-research/rune","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apple/ml-reed","reach":null}],"summary":{"ran":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"a29809de3135626d","entry":"EnvironmentContrastiveBatch","repo":"apple/ml-reed","repo_kind":"listed","path":"reed/models/self_predictive_representations_model.py","file_url":"https://github.com/apple/ml-reed/blob/HEAD/reed/models/self_predictive_representations_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a29809de3135626d"}},{"code_sha256_prefix":"b518ac768ca194c9","entry":"RewardModel","repo":"rll-research/rune","repo_kind":"listed","path":"reward_model_explore.py","file_url":"https://github.com/rll-research/rune/blob/HEAD/reward_model_explore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b518ac768ca194c9"}},{"code_sha256_prefix":"b64a58c0dd687c8c","entry":"StateActionSelfPredictiveRepresentationsNetworkEnsemble","repo":"apple/ml-reed","repo_kind":"listed","path":"reed/models/self_predictive_representations_model.py","file_url":"https://github.com/apple/ml-reed/blob/HEAD/reed/models/self_predictive_representations_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b64a58c0dd687c8c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}