{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-goal-conditioned-rl-algorithms","title":"Accelerating Goal-Conditioned RL Algorithms and Research","arxiv_id":"2408.11052","date":"2024-08-20","proceeding":null,"authors":["Michał Bortkiewicz","Władysław Pałucki","Vivek Myers","Tadeusz Dziarmaga","Tomasz Arczewski","Łukasz Kuciński","Benjamin Eysenbach"],"abstract":"Self-supervision has the potential to transform reinforcement learning (RL), paralleling the breakthroughs it has enabled in other areas of machine learning. While self-supervised learning in other domains aims to find patterns in a fixed dataset, self-supervised goal-conditioned reinforcement learning (GCRL) agents discover new behaviors by learning from the goals achieved during unstructured interaction with the environment. However, these methods have failed to see similar success, both due to a lack of data from slow environment simulations as well as a lack of stable algorithms. We take a step toward addressing both of these issues by releasing a high-performance codebase and benchmark (JaxGCRL) for self-supervised GCRL, enabling researchers to train agents for millions of environment steps in minutes on a single GPU. By utilizing GPU-accelerated replay buffers, environments, and a stable contrastive RL algorithm, we reduce training time by up to $22\\times$. Additionally, we assess key design choices in contrastive RL, identifying those that most effectively stabilize and enhance training performance. With this approach, we provide a foundation for future research in self-supervised GCRL, enabling researchers to quickly iterate on new ideas and evaluate them in diverse and challenging environments. Website + Code: https://github.com/MichalBortkiewicz/JaxGCRL","url_abs":"https://arxiv.org/abs/2408.11052v3","url_pdf":"https://arxiv.org/pdf/2408.11052v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-goal-conditioned-rl-algorithms","repo_url":"https://github.com/michalbortkiewicz/jaxgcrl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.11052","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11052"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MichalBortkiewicz/JaxGCRL","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/michalbortkiewicz/jaxgcrl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e848a7e1eecab2c4","entry":"apply_logit_scale","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/agents/crl/losses.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/agents/crl/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e848a7e1eecab2c4"}},{"code_sha256_prefix":"ba21bcd0d0467788","entry":"contrastive_loss_fn","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/agents/crl/losses.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/agents/crl/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba21bcd0d0467788"}},{"code_sha256_prefix":"d2c4953957f1533f","entry":"energy_fn","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/agents/crl/losses.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/agents/crl/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2c4953957f1533f"}},{"code_sha256_prefix":"55de8ec7d2fb9e0e","entry":"find","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/envs/ant_ball_maze.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/envs/ant_ball_maze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55de8ec7d2fb9e0e"}},{"code_sha256_prefix":"12acaee1021d466e","entry":"find_goals","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/envs/ant_maze.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/envs/ant_maze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12acaee1021d466e"}},{"code_sha256_prefix":"8acce921c5f5d9f6","entry":"find_starts","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/envs/ant_maze.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/envs/ant_maze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8acce921c5f5d9f6"}},{"code_sha256_prefix":"e9ec954a0a71c3f3","entry":"flatten_batch","repo":"MichalBortkiewicz/JaxGCRL","repo_kind":"official","path":"jaxgcrl/agents/crl/crl.py","file_url":"https://github.com/MichalBortkiewicz/JaxGCRL/blob/HEAD/jaxgcrl/agents/crl/crl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9ec954a0a71c3f3"}},{"code_sha256_prefix":"b9fed04b3ce8f205","entry":"make_maze","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/envs/ant_ball_maze.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/envs/ant_ball_maze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9fed04b3ce8f205"}},{"code_sha256_prefix":"c4df795fab23a2d9","entry":"make_maze","repo":"michalbortkiewicz/jaxgcrl","repo_kind":"official","path":"jaxgcrl/envs/ant_maze.py","file_url":"https://github.com/michalbortkiewicz/jaxgcrl/blob/HEAD/jaxgcrl/envs/ant_maze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c4df795fab23a2d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}