{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-aware-reward-free-exploration","title":"Uncertainty-Aware Reward-Free Exploration with General Function Approximation","arxiv_id":"2406.16255","date":"2024-06-24","proceeding":null,"authors":["Junkai Zhang","Weitong Zhang","Dongruo Zhou","Quanquan Gu"],"abstract":"Mastering multiple tasks through exploration and learning in an environment poses a significant challenge in reinforcement learning (RL). Unsupervised RL has been introduced to address this challenge by training policies with intrinsic rewards rather than extrinsic rewards. However, current intrinsic reward designs and unsupervised RL algorithms often overlook the heterogeneous nature of collected samples, thereby diminishing their sample efficiency. To overcome this limitation, in this paper, we propose a reward-free RL algorithm called \\alg. The key idea behind our algorithm is an uncertainty-aware intrinsic reward for exploring the environment and an uncertainty-weighted learning process to handle heterogeneous uncertainty in different samples. Theoretically, we show that in order to find an $\\epsilon$-optimal policy, GFA-RFE needs to collect $\\tilde{O} (H^2 \\log N_{\\mathcal F} (\\epsilon) \\mathrm{dim} (\\mathcal F) / \\epsilon^2 )$ number of episodes, where $\\mathcal F$ is the value function class with covering number $N_{\\mathcal F} (\\epsilon)$ and generalized eluder dimension $\\mathrm{dim} (\\mathcal F)$. Such a result outperforms all existing reward-free RL algorithms. We further implement and evaluate GFA-RFE across various domains and tasks in the DeepMind Control Suite. Experiment results show that GFA-RFE outperforms or is comparable to the performance of state-of-the-art unsupervised RL algorithms.","url_abs":"https://arxiv.org/abs/2406.16255v2","url_pdf":"https://arxiv.org/pdf/2406.16255v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-aware-reward-free-exploration","repo_url":"https://github.com/uclaml/gfa-rfe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16255","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16255"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/uclaml/GFA-RFE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uclaml/gfa-rfe","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_honours":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"419af119bd09b0ac","entry":"Actor","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"419af119bd09b0ac"}},{"code_sha256_prefix":"790cb19787a27e3c","entry":"Critic","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"790cb19787a27e3c"}},{"code_sha256_prefix":"2e578a8124e51723","entry":"Encoder","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e578a8124e51723"}},{"code_sha256_prefix":"5aee04fb3686517a","entry":"RandomShiftsAug","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aee04fb3686517a"}},{"code_sha256_prefix":"a8a8005cfb1d6a98","entry":"VectorizedCritics","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8a8005cfb1d6a98"}},{"code_sha256_prefix":"9f840a22d31a89bc","entry":"episode_len","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f840a22d31a89bc"}},{"code_sha256_prefix":"2478bb95b38ef24f","entry":"grad_norm","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2478bb95b38ef24f"}},{"code_sha256_prefix":"184876e5f4ab6562","entry":"load_episode","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"184876e5f4ab6562"}},{"code_sha256_prefix":"e4f44028e6834d07","entry":"make_replay_loader","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"replay_buffer.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f44028e6834d07"}},{"code_sha256_prefix":"8708aaca52006e47","entry":"param_norm","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8708aaca52006e47"}},{"code_sha256_prefix":"ae234269debed3f4","entry":"schedule","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae234269debed3f4"}},{"code_sha256_prefix":"6244e9922f4e0610","entry":"to_torch","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6244e9922f4e0610"}},{"code_sha256_prefix":"dbe65af46016bc97","entry":"D2Agent","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbe65af46016bc97"}},{"code_sha256_prefix":"4c262c61dad0f383","entry":"DDPGAgent","repo":"uclaml/gfa-rfe","repo_kind":"official","path":"agent/dsquare.py","file_url":"https://github.com/uclaml/gfa-rfe/blob/HEAD/agent/dsquare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c262c61dad0f383"}},{"code_sha256_prefix":"2a76dfe317adeb03","entry":"make_agent","repo":"uclaml/GFA-RFE","repo_kind":"official","path":"offline.py","file_url":"https://github.com/uclaml/GFA-RFE/blob/HEAD/offline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a76dfe317adeb03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}