{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/remember-and-forget-for-experience-replay","title":"Remember and Forget for Experience Replay","arxiv_id":"1807.05827","date":"2018-07-16","proceeding":"ICLR 2019 5","authors":["Guido Novati","Petros Koumoutsakos"],"abstract":"Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors and can undermine the performance of ER. Many algorithms mitigate this issue by tuning hyper-parameters to slow down policy changes. An alternative is to actively enforce the similarity between policy and the experiences in the replay memory. We introduce Remember and Forget Experience Replay (ReF-ER), a novel method that can enhance RL algorithms with parameterized policies. ReF-ER (1) skips gradients computed from experiences that are too unlikely with the current policy and (2) regulates policy changes within a trust region of the replayed behaviors. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods. We find that ReF-ER consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.","url_abs":"https://arxiv.org/abs/1807.05827v4","url_pdf":"https://arxiv.org/pdf/1807.05827v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"remember-and-forget-for-experience-replay","repo_url":"https://github.com/cselab/smarties","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"remember-and-forget-for-experience-replay","repo_url":"https://gitlab.com/stchrist/ViscousThreads","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1807.05827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1807.05827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/stchrist/ViscousThreads","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cselab/smarties","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"95812260f5c7d8bd","entry":"getWorkerRankToPlot","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties_plot_rew.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties_plot_rew.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95812260f5c7d8bd"}},{"code_sha256_prefix":"7ecb8c12385798c3","entry":"is_exe","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ecb8c12385798c3"}},{"code_sha256_prefix":"e7900b463dd6c7e4","entry":"nameAxis","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties_plot_obs.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties_plot_obs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7900b463dd6c7e4"}},{"code_sha256_prefix":"4d556262f39cbb51","entry":"plotReward","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties_plot_rew.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties_plot_rew.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d556262f39cbb51"}},{"code_sha256_prefix":"1b79c055d7cca545","entry":"setEnvironmentFlags","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b79c055d7cca545"}},{"code_sha256_prefix":"bb243d8b39c7f796","entry":"setTrainOrEvaluate","repo":"cselab/smarties","repo_kind":"official","path":"bin/smarties.py","file_url":"https://github.com/cselab/smarties/blob/HEAD/bin/smarties.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb243d8b39c7f796"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}