{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rank-the-episodes-a-simple-approach-for-1","title":"Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments","arxiv_id":"2101.08152","date":"2021-01-20","proceeding":"ICLR 2021 1","authors":["Daochen Zha","Wenye Ma","Lei Yuan","Xia Hu","Ji Liu"],"abstract":"Exploration under sparse reward is a long-standing challenge of model-free reinforcement learning. The state-of-the-art methods address this challenge by introducing intrinsic rewards to encourage exploration in novel states or uncertain environment dynamics. Unfortunately, methods based on intrinsic rewards often fall short in procedurally-generated environments, where a different environment is generated in each episode so that the agent is not likely to visit the same state more than once. Motivated by how humans distinguish good exploration behaviors by looking into the entire episode, we introduce RAPID, a simple yet effective episode-level exploration method for procedurally-generated environments. RAPID regards each episode as a whole and gives an episodic exploration score from both per-episode and long-term views. Those highly scored episodes are treated as good exploration behaviors and are stored in a small ranking buffer. The agent then imitates the episodes in the buffer to reproduce the past good exploration behaviors. We demonstrate our method on several procedurally-generated MiniGrid environments, a first-person-view 3D Maze navigation task from MiniWorld, and several sparse MuJoCo tasks. The results show that RAPID significantly outperforms the state-of-the-art intrinsic reward strategies in terms of sample efficiency and final performance. The code is available at https://github.com/daochenzha/rapid","url_abs":"https://arxiv.org/abs/2101.08152v2","url_pdf":"https://arxiv.org/pdf/2101.08152v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rank-the-episodes-a-simple-approach-for-1","repo_url":"https://github.com/daochenzha/rapid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"rank-the-episodes-a-simple-approach-for-1","repo_url":"https://github.com/maximecb/gym-miniworld","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"rank-the-episodes-a-simple-approach-for-1","repo_url":"https://github.com/aklein1995/exploration_sil_im","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2101.08152","atlas_url":"https://app.syntology.ai/?focus=2101.08152","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.08152"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/daochenzha/rapid","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aklein1995/exploration_sil_im","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maximecb/gym-miniworld","reach":{"status":"unanswered"}}],"summary":{"ran":9,"ran_draft_wrong":2,"ran_fixture":2,"ran_violates":1,"unverified":14},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"listed":{"samples":23,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ad5584bf82eab7fe","entry":"BeBold","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad5584bf82eab7fe"}},{"code_sha256_prefix":"978c689ae69157a3","entry":"Counter","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"978c689ae69157a3"}},{"code_sha256_prefix":"d6eaff6eabe8943b","entry":"Counter","repo":"daochenzha/rapid","repo_kind":"official","path":"rapid/buffer.py","file_url":"https://github.com/daochenzha/rapid/blob/HEAD/rapid/buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6eaff6eabe8943b"}},{"code_sha256_prefix":"242a79b49bc8f5de","entry":"HumanOutputFormat","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"242a79b49bc8f5de"}},{"code_sha256_prefix":"081800d1f215fd82","entry":"JSONOutputFormat","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"081800d1f215fd82"}},{"code_sha256_prefix":"ecaf9ce3efa32820","entry":"Logger","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecaf9ce3efa32820"}},{"code_sha256_prefix":"8758308dc6c0727f","entry":"constfn","repo":"daochenzha/rapid","repo_kind":"official","path":"rapid/agent.py","file_url":"https://github.com/daochenzha/rapid/blob/HEAD/rapid/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8758308dc6c0727f"}},{"code_sha256_prefix":"6f9580f36835d52d","entry":"explained_variance","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f9580f36835d52d"}},{"code_sha256_prefix":"f0b5e724afa08592","entry":"get_current","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0b5e724afa08592"}},{"code_sha256_prefix":"19fa70dab4a7056f","entry":"get_dir","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19fa70dab4a7056f"}},{"code_sha256_prefix":"1bd23d9c5df7763c","entry":"get_local_bonus","repo":"daochenzha/rapid","repo_kind":"official","path":"rapid/buffer.py","file_url":"https://github.com/daochenzha/rapid/blob/HEAD/rapid/buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bd23d9c5df7763c"}},{"code_sha256_prefix":"3c4301eba0f1aaf2","entry":"get_rank_without_mpi_import","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c4301eba0f1aaf2"}},{"code_sha256_prefix":"8b7334946d2ed532","entry":"make_output_format","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b7334946d2ed532"}},{"code_sha256_prefix":"baf2a28d376d7803","entry":"sf01","repo":"daochenzha/rapid","repo_kind":"official","path":"rapid/agent.py","file_url":"https://github.com/daochenzha/rapid/blob/HEAD/rapid/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"baf2a28d376d7803"}},{"code_sha256_prefix":"b21a3fc9290718d4","entry":"AbstractEnvRunner","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b21a3fc9290718d4"}},{"code_sha256_prefix":"0c58056d576d9ba8","entry":"CSVOutputFormat","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c58056d576d9ba8"}},{"code_sha256_prefix":"efd67c7b09262d14","entry":"Model","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd67c7b09262d14"}},{"code_sha256_prefix":"c0f83827d6c75fca","entry":"RankingBuffer","repo":"daochenzha/rapid","repo_kind":"official","path":"rapid/buffer.py","file_url":"https://github.com/daochenzha/rapid/blob/HEAD/rapid/buffer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0f83827d6c75fca"}},{"code_sha256_prefix":"b18fdb2df2966f99","entry":"Runner","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b18fdb2df2966f99"}},{"code_sha256_prefix":"07d722b0c7f269f6","entry":"TensorBoardOutputFormat","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07d722b0c7f269f6"}},{"code_sha256_prefix":"5b226f8121d5718b","entry":"_configure_default_logger","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b226f8121d5718b"}},{"code_sha256_prefix":"819a51a05cef8459","entry":"configure","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"819a51a05cef8459"}},{"code_sha256_prefix":"b5664e29a5753d3d","entry":"info","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5664e29a5753d3d"}},{"code_sha256_prefix":"e22453516d7e1e51","entry":"learn_rapid","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e22453516d7e1e51"}},{"code_sha256_prefix":"13289659b2b68f4f","entry":"limit_cuda_visible_devices","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13289659b2b68f4f"}},{"code_sha256_prefix":"420a7935bc54a351","entry":"log","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"420a7935bc54a351"}},{"code_sha256_prefix":"8db9834e56303b87","entry":"make_session","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8db9834e56303b87"}},{"code_sha256_prefix":"cbb527600328539c","entry":"set_global_seeds","repo":"aklein1995/exploration_sil_im","repo_kind":"listed","path":"rapid/rapid_agent.py","file_url":"https://github.com/aklein1995/exploration_sil_im/blob/HEAD/rapid/rapid_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbb527600328539c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}