{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-batch-experience-replay","title":"Large Batch Experience Replay","arxiv_id":"2110.01528","date":"2021-10-04","proceeding":null,"authors":["Thibault Lahire","Matthieu Geist","Emmanuel Rachelson"],"abstract":"Several algorithms have been proposed to sample non-uniformly the replay buffer of deep Reinforcement Learning (RL) agents to speed-up learning, but very few theoretical foundations of these sampling schemes have been provided. Among others, Prioritized Experience Replay appears as a hyperparameter sensitive heuristic, even though it can provide good performance. In this work, we cast the replay buffer sampling problem as an importance sampling one for estimating the gradient. This allows deriving the theoretically optimal sampling distribution, yielding the best theoretical convergence speed. Elaborating on the knowledge of the ideal sampling scheme, we exhibit new theoretical foundations of Prioritized Experience Replay. The optimal sampling distribution being intractable, we make several approximations providing good results in practice and introduce, among others, LaBER (Large Batch Experience Replay), an easy-to-code and efficient method for sampling the replay buffer. LaBER, which can be combined with Deep Q-Networks, distributional RL agents or actor-critic methods, yields improved performance over a diverse range of Atari games and PyBullet environments, compared to the base agent it is implemented on and to other prioritization schemes.","url_abs":"https://arxiv.org/abs/2110.01528v2","url_pdf":"https://arxiv.org/pdf/2110.01528v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-batch-experience-replay","repo_url":"https://github.com/sureli/laber","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"large-batch-experience-replay","repo_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"prioritized-experience-replay","method_name":"Prioritized Experience Replay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2110.01528","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.01528"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sureli/laber","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay","reach":null}],"summary":{"ran":4,"ran_fixture":2,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1},"listed":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"508f079e9d0ccf91","entry":"Actor","repo":"sureli/laber","repo_kind":"official","path":"LaBER/continuous/LABER_SAC.py","file_url":"https://github.com/sureli/laber/blob/HEAD/LaBER/continuous/LABER_SAC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"508f079e9d0ccf91"}},{"code_sha256_prefix":"f9a1b229fd3fdc31","entry":"Batch","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9a1b229fd3fdc31"}},{"code_sha256_prefix":"4949a32e6cb9c3c5","entry":"Critic","repo":"sureli/laber","repo_kind":"official","path":"LaBER/continuous/LABER_SAC.py","file_url":"https://github.com/sureli/laber/blob/HEAD/LaBER/continuous/LABER_SAC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4949a32e6cb9c3c5"}},{"code_sha256_prefix":"4fe1a6b8253ee0a5","entry":"LABER_SAC","repo":"sureli/laber","repo_kind":"official","path":"LaBER/continuous/LABER_SAC.py","file_url":"https://github.com/sureli/laber/blob/HEAD/LaBER/continuous/LABER_SAC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fe1a6b8253ee0a5"}},{"code_sha256_prefix":"acd5bcd47c0df9f6","entry":"get_state","repo":"sureli/laber","repo_kind":"official","path":"MinAtar_experiments/agents/dqn_LABER.py","file_url":"https://github.com/sureli/laber/blob/HEAD/MinAtar_experiments/agents/dqn_LABER.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acd5bcd47c0df9f6"}},{"code_sha256_prefix":"7f72e8b477521aba","entry":"grad_norm","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f72e8b477521aba"}},{"code_sha256_prefix":"a6eed9bf6f077596","entry":"world_dynamics","repo":"sureli/laber","repo_kind":"official","path":"MinAtar_experiments/agents/dqn_LABER.py","file_url":"https://github.com/sureli/laber/blob/HEAD/MinAtar_experiments/agents/dqn_LABER.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6eed9bf6f077596"}},{"code_sha256_prefix":"a02561f614036415","entry":"Model","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a02561f614036415"}},{"code_sha256_prefix":"406850d16892ed44","entry":"_update_jit_laber","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"406850d16892ed44"}},{"code_sha256_prefix":"e85be23fbac3d092","entry":"target_update","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e85be23fbac3d092"}},{"code_sha256_prefix":"33815b2ad6fcd3d0","entry":"update","repo":"xavierchanglingli/regularized-optimal-experience-replay","repo_kind":"listed","path":"sac_learner.py","file_url":"https://github.com/xavierchanglingli/regularized-optimal-experience-replay/blob/HEAD/sac_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33815b2ad6fcd3d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}