{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hindsight-experience-replay","title":"Hindsight Experience Replay","arxiv_id":"1707.01495","date":"2017-07-05","proceeding":"NeurIPS 2017 12","authors":["Marcin Andrychowicz","Filip Wolski","Alex Ray","Jonas Schneider","Rachel Fong","Peter Welinder","Bob McGrew","Josh Tobin","Pieter Abbeel","Wojciech Zaremba"],"abstract":"Dealing with sparse rewards is one of the biggest challenges in Reinforcement\nLearning (RL). We present a novel technique called Hindsight Experience Replay\nwhich allows sample-efficient learning from rewards which are sparse and binary\nand therefore avoid the need for complicated reward engineering. It can be\ncombined with an arbitrary off-policy RL algorithm and may be seen as a form of\nimplicit curriculum.\n  We demonstrate our approach on the task of manipulating objects with a\nrobotic arm. In particular, we run experiments on three different tasks:\npushing, sliding, and pick-and-place, in each case using only binary rewards\nindicating whether or not the task is completed. Our ablation studies show that\nHindsight Experience Replay is a crucial ingredient which makes training\npossible in these challenging environments. We show that our policies trained\non a physics simulation can be deployed on a physical robot and successfully\ncomplete the task.","url_abs":"http://arxiv.org/abs/1707.01495v3","url_pdf":"http://arxiv.org/pdf/1707.01495v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/AdamStelmaszczyk/dqn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/Baichenjia/BHER","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/JunkyByte/HER_DQN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/Medabid1/RL_Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/Stevenhunter167/HER_Minigrid","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/TianhongDai/hindsight-experience-replay","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/YangRui2015/Modular_HER","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/bacdavid/HER","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/ethancaballero/neural-engineers-first-attempt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/fapont/hackaton-hiparis-2021","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/flowersteam/curious","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/hemilpanchiwala/Hindsight-Experience-Replay","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/hietalajulius/clothmanip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/hietalajulius/dynamic-cloth-folding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/jaem-seo/AI_tokamak_control","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/jangirrishabh/Overcoming-exploration-from-demos","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/mehdimashayekhi/Some-RL-Implementation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/mindspore-courses/Deep-Reinforcement-Learning-Algorithms-with-MindSpore","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/offworld-projects/research-halgan","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/rk1998/robot-sac","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/sdicastro/simandrealbettertogether","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/sjYoondeltar/IQN_example","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/sjYoondeltar/myRL_example","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/sumitsk/HER","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/tenvinc/CS4246-Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/DLR-RM/stable-baselines3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/hill-a/stable-baselines","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"hindsight-experience-replay","repo_url":"https://github.com/opendilab/DI-engine/blob/main/ding/reward_model/her_reward_model.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.01495","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1707.01495"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hill-a/stable-baselines","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DLR-RM/stable-baselines3","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AdamStelmaszczyk/dqn","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YangRui2015/Modular_HER","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/DI-engine/blob/main/ding/reward_model/her_reward_model.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sumitsk/HER","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JunkyByte/HER_DQN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjYoondeltar/myRL_example","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/Deep-Reinforcement-Learning-Algorithms-with-MindSpore","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hietalajulius/dynamic-cloth-folding","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mehdimashayekhi/Some-RL-Implementation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hietalajulius/clothmanip","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fapont/hackaton-hiparis-2021","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjYoondeltar/IQN_example","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TianhongDai/hindsight-experience-replay","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Medabid1/RL_Project","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rk1998/robot-sac","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bacdavid/HER","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flowersteam/curious","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Stevenhunter167/HER_Minigrid","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Baichenjia/BHER","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/offworld-projects/research-halgan","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jangirrishabh/Overcoming-exploration-from-demos","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ethancaballero/neural-engineers-first-attempt","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hemilpanchiwala/Hindsight-Experience-Replay","reach":null}],"summary":{"ran":10,"ran_honours":1,"ran_draft_wrong":4,"ran_fixture":1},"by_repo_kind":{"listed":{"samples":15,"ran":15,"repositories":12}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"a2d77ddd5e9cb248","entry":"HER","repo":"bacdavid/HER","repo_kind":"listed","path":"hindsight_experience.py","file_url":"https://github.com/bacdavid/HER/blob/HEAD/hindsight_experience.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2d77ddd5e9cb248"}},{"code_sha256_prefix":"9d20e8525bae5d5c","entry":"HER","repo":"JunkyByte/HER_DQN","repo_kind":"listed","path":"src/her_module/her.py","file_url":"https://github.com/JunkyByte/HER_DQN/blob/HEAD/src/her_module/her.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d20e8525bae5d5c"}},{"code_sha256_prefix":"e959db3969fc9191","entry":"HER_Sampler","repo":"YangRui2015/Modular_HER","repo_kind":"listed","path":"mher/samplers/her_sampler.py","file_url":"https://github.com/YangRui2015/Modular_HER/blob/HEAD/mher/samplers/her_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e959db3969fc9191"}},{"code_sha256_prefix":"f0fd2810588c0760","entry":"HER_sampler","repo":"sumitsk/HER","repo_kind":"listed","path":"her.py","file_url":"https://github.com/sumitsk/HER/blob/HEAD/her.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0fd2810588c0760"}},{"code_sha256_prefix":"06e4f2c26ad752ba","entry":"HerRewardModel","repo":"opendilab/DI-engine","repo_kind":"listed","path":"ding/reward_model/her_reward_model.py","file_url":"https://github.com/opendilab/DI-engine/blob/HEAD/ding/reward_model/her_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06e4f2c26ad752ba"}},{"code_sha256_prefix":"cee4c46cd21980ff","entry":"HindsightExperienceReplayMemory","repo":"hemilpanchiwala/Hindsight-Experience-Replay","repo_kind":"listed","path":"dqn_with_her/HERMemory.py","file_url":"https://github.com/hemilpanchiwala/Hindsight-Experience-Replay/blob/HEAD/dqn_with_her/HERMemory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cee4c46cd21980ff"}},{"code_sha256_prefix":"5b95feb61a547c1b","entry":"RelabelSampler","repo":"YangRui2015/Modular_HER","repo_kind":"listed","path":"mher/samplers/her_sampler.py","file_url":"https://github.com/YangRui2015/Modular_HER/blob/HEAD/mher/samplers/her_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b95feb61a547c1b"}},{"code_sha256_prefix":"8c4664b53d0576fa","entry":"Sampler","repo":"YangRui2015/Modular_HER","repo_kind":"listed","path":"mher/samplers/her_sampler.py","file_url":"https://github.com/YangRui2015/Modular_HER/blob/HEAD/mher/samplers/her_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c4664b53d0576fa"}},{"code_sha256_prefix":"f616d3cbede3329d","entry":"concat","repo":"bacdavid/HER","repo_kind":"listed","path":"hindsight_experience.py","file_url":"https://github.com/bacdavid/HER/blob/HEAD/hindsight_experience.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f616d3cbede3329d"}},{"code_sha256_prefix":"40295510a8fb1d25","entry":"get_replay_buffer_kwargs","repo":"hietalajulius/dynamic-cloth-folding","repo_kind":"listed","path":"utils/general_utils.py","file_url":"https://github.com/hietalajulius/dynamic-cloth-folding/blob/HEAD/utils/general_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40295510a8fb1d25"}},{"code_sha256_prefix":"18eb3a56918d44a6","entry":"her_sample","repo":"tenvinc/CS4246-Project","repo_kind":"listed","path":"agent/core.py","file_url":"https://github.com/tenvinc/CS4246-Project/blob/HEAD/agent/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18eb3a56918d44a6"}},{"code_sha256_prefix":"6001bd605e7c8a38","entry":"her_sampler","repo":"sdicastro/simandrealbettertogether","repo_kind":"listed","path":"her_modules/her.py","file_url":"https://github.com/sdicastro/simandrealbettertogether/blob/HEAD/her_modules/her.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6001bd605e7c8a38"}},{"code_sha256_prefix":"732ee2071c58239b","entry":"her_sampler","repo":"Medabid1/RL_Project","repo_kind":"listed","path":"utils/her.py","file_url":"https://github.com/Medabid1/RL_Project/blob/HEAD/utils/her.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"732ee2071c58239b"}},{"code_sha256_prefix":"e0785843af667cdd","entry":"make_sample_her_transitions","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e0785843af667cdd"}},{"code_sha256_prefix":"14bbcdb9ee8b58d7","entry":"make_sample_her_transitions","repo":"Baichenjia/BHER","repo_kind":"listed","path":"BHER/bher/her.py","file_url":"https://github.com/Baichenjia/BHER/blob/HEAD/BHER/bher/her.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14bbcdb9ee8b58d7"}},{"code_sha256_prefix":"dc602be819c160dc","entry":"make_sample_her_transitions","repo":"flowersteam/curious","repo_kind":"listed","path":"baselines/her/her.py","file_url":"https://github.com/flowersteam/curious/blob/HEAD/baselines/her/her.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc602be819c160dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}