{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/first-return-then-explore","title":"First return, then explore","arxiv_id":"2004.12919","date":"2020-04-27","proceeding":null,"authors":["Adrien Ecoffet","Joost Huizinga","Joel Lehman","Kenneth O. Stanley","Jeff Clune"],"abstract":"The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states (\"detachment\") and from failing to first return to a state before exploring from it (\"derailment\"). We introduce Go-Explore, a family of algorithms that addresses these two challenges directly through the simple principles of explicitly remembering promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games, with orders of magnitude improvements on the grand challenges Montezuma's Revenge and Pitfall. We also demonstrate the practical potential of Go-Explore on a sparse-reward pick-and-place robotics task. Additionally, we show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handle stochasticity throughout training. The substantial performance gains from Go-Explore suggest that the simple principles of remembering states, returning to them, and exploring from them are a powerful and general approach to exploration, an insight that may prove critical to the creation of truly intelligent learning agents.","url_abs":"https://arxiv.org/abs/2004.12919v6","url_pdf":"https://arxiv.org/pdf/2004.12919v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"first-return-then-explore","repo_url":"https://github.com/uber-research/go-explore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"first-return-then-explore","repo_url":"https://github.com/qgallouedec/lge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"montezumas-revenge","task_name":"Montezuma's Revenge"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"go-explore","method_name":"Go-Explore"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-2600-berzerk","task":"Atari Games","dataset":"Atari 2600 Berzerk","model":"Go-Explore","rank_in_archive_order":1,"of":39,"metrics":{"Score":"197376"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-bowling","task":"Atari Games","dataset":"Atari 2600 Bowling","model":"Go-Explore","rank_in_archive_order":2,"of":44,"metrics":{"Score":"260"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-centipede","task":"Atari Games","dataset":"Atari 2600 Centipede","model":"Go-Explore","rank_in_archive_order":1,"of":45,"metrics":{"Score":"1422628"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-freeway","task":"Atari Games","dataset":"Atari 2600 Freeway","model":"Go-Explore","rank_in_archive_order":5,"of":59,"metrics":{"Score":"34"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-gravitar","task":"Atari Games","dataset":"Atari 2600 Gravitar","model":"Go-Explore","rank_in_archive_order":4,"of":53,"metrics":{"Score":"7588"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-montezumas-revenge","task":"Atari Games","dataset":"Atari 2600 Montezuma's Revenge","model":"Go-Explore","rank_in_archive_order":1,"of":50,"metrics":{"Score":"43791"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-pitfall","task":"Atari Games","dataset":"Atari 2600 Pitfall!","model":"Go-Explore","rank_in_archive_order":3,"of":23,"metrics":{"Score":"6954"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-private-eye","task":"Atari Games","dataset":"Atari 2600 Private Eye","model":"Go-Explore","rank_in_archive_order":1,"of":52,"metrics":{"Score":"95756"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-skiing","task":"Atari Games","dataset":"Atari 2600 Skiing","model":"Go-Explore","rank_in_archive_order":13,"of":23,"metrics":{"Score":"-3660"},"uses_additional_data":true},{"leaderboard":"/sota/atari-games-on-atari-2600-solaris","task":"Atari Games","dataset":"Atari 2600 Solaris","model":"Go-Explore","rank_in_archive_order":2,"of":23,"metrics":{"Score":"19671"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-venture","task":"Atari Games","dataset":"Atari 2600 Venture","model":"Go-Explore","rank_in_archive_order":2,"of":55,"metrics":{"Score":"2281"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-games","task":"Atari Games","dataset":"Atari games","model":"Go-Explore","rank_in_archive_order":4,"of":12,"metrics":{"Mean Human Normalized Score":"4989.94%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.12919","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.12919"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qgallouedec/lge","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uber-research/go-explore","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"7aca6d4823baaf95","entry":"getsize","repo":"uber-research/go-explore","repo_kind":"official","path":"policy_based/goexplore_py/goexplore.py","file_url":"https://github.com/uber-research/go-explore/blob/HEAD/policy_based/goexplore_py/goexplore.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7aca6d4823baaf95"}},{"code_sha256_prefix":"efd5de4203794fee","entry":"seed_pool_wrapper","repo":"uber-research/go-explore","repo_kind":"official","path":"robustified/goexplore_py/goexplore.py","file_url":"https://github.com/uber-research/go-explore/blob/HEAD/robustified/goexplore_py/goexplore.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"efd5de4203794fee"}},{"code_sha256_prefix":"c0b41660051f8848","entry":"get_downscale","repo":"uber-research/go-explore","repo_kind":"official","path":"robustified/goexplore_py/goexplore.py","file_url":"https://github.com/uber-research/go-explore/blob/HEAD/robustified/goexplore_py/goexplore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c0b41660051f8848"}},{"code_sha256_prefix":"7f626c87007c22d4","entry":"run_f_seeded","repo":"uber-research/go-explore","repo_kind":"official","path":"robustified/goexplore_py/goexplore.py","file_url":"https://github.com/uber-research/go-explore/blob/HEAD/robustified/goexplore_py/goexplore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7f626c87007c22d4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}