{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-exploration-with-unlabeled-prior-1","title":"Accelerating Exploration with Unlabeled Prior Data","arxiv_id":"2311.05067","date":"2023-11-09","proceeding":"NeurIPS 2023 11","authors":["Qiyang Li","Jason Zhang","Dibya Ghosh","Amy Zhang","Sergey Levine"],"abstract":"Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms. However, in the real world, agents rarely need to solve sparse reward tasks entirely from scratch. More often, we might possess prior experience to draw on that provides considerable guidance about which actions and outcomes are possible in the world, which we can use to explore more effectively for new tasks. In this work, we study how prior data without reward labels may be used to guide and accelerate exploration for an agent solving a new sparse reward task. We propose a simple approach that learns a reward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization. This general formula leads to rapid exploration in several challenging sparse-reward domains where tabula rasa exploration is insufficient, including the AntMaze domain, Adroit hand manipulation domain, and a visual simulated robotic manipulation domain. Our results highlight the ease of incorporating unlabeled prior data into existing online RL algorithms, and the (perhaps surprising) effectiveness of doing so.","url_abs":"https://arxiv.org/abs/2311.05067v2","url_pdf":"https://arxiv.org/pdf/2311.05067v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-exploration-with-unlabeled-prior-1","repo_url":"https://github.com/facebookresearch/explore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.05067","atlas_url":"https://app.syntology.ai/?focus=2311.05067","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.05067"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ikostrikov/rlpd","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/avisingh599/cog","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/explore","reach":null}],"summary":{"ran_draft_wrong":2,"ran":8,"ran_fixture":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1},"found_in_text":{"samples":8,"ran":7,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"089275b7fa4d5916","entry":"add_prefix","repo":"facebookresearch/explore","repo_kind":"official","path":"train_finetuning.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/train_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"089275b7fa4d5916"}},{"code_sha256_prefix":"bbbf2d644361bb77","entry":"MLP","repo":"facebookresearch/explore","repo_kind":"official","path":"rlpd/agents/rm.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/rlpd/agents/rm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bbbf2d644361bb77"}},{"code_sha256_prefix":"31328481902ec81d","entry":"StateActionValue","repo":"facebookresearch/explore","repo_kind":"official","path":"rlpd/agents/rm.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/rlpd/agents/rm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"31328481902ec81d"}},{"code_sha256_prefix":"020d4443bd958d4c","entry":"check_overlap","repo":"facebookresearch/explore","repo_kind":"official","path":"train_finetuning.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/train_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"020d4443bd958d4c"}},{"code_sha256_prefix":"93d28ca67303e04b","entry":"clip_magnitude","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/pythonplusplus.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93d28ca67303e04b"}},{"code_sha256_prefix":"7a2c5b4e604fd307","entry":"combine","repo":"facebookresearch/explore","repo_kind":"official","path":"train_finetuning.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/train_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7a2c5b4e604fd307"}},{"code_sha256_prefix":"fadf5f762ff04cd6","entry":"create_stats_ordered_dict","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/core/eval_util.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fadf5f762ff04cd6"}},{"code_sha256_prefix":"a5d10a0b704c8db7","entry":"get_average_returns","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/core/eval_util.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5d10a0b704c8db7"}},{"code_sha256_prefix":"9b9a51e3e3be7a6e","entry":"get_flat_params","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/core/batch_rl_algorithm.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/core/batch_rl_algorithm.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b9a51e3e3be7a6e"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/pythonplusplus.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"4138766e854db009","entry":"is_numeric","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/pythonplusplus.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4138766e854db009"}},{"code_sha256_prefix":"5cb0c55f58279f59","entry":"simple_separated_format","repo":"avisingh599/cog","repo_kind":"found_in_text","path":"rlkit/core/tabulate.py","file_url":"https://github.com/avisingh599/cog/blob/HEAD/rlkit/core/tabulate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cb0c55f58279f59"}},{"code_sha256_prefix":"7b4b28cd176c0e5a","entry":"RM","repo":"facebookresearch/explore","repo_kind":"official","path":"rlpd/agents/rm.py","file_url":"https://github.com/facebookresearch/explore/blob/HEAD/rlpd/agents/rm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7b4b28cd176c0e5a"}},{"code_sha256_prefix":"c021b9fd8bb7677d","entry":"subsample_ensemble","repo":"ikostrikov/rlpd","repo_kind":"found_in_text","path":"rlpd/networks/ensemble.py","file_url":"https://github.com/ikostrikov/rlpd/blob/HEAD/rlpd/networks/ensemble.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c021b9fd8bb7677d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}