{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-does-return-conditioned-supervised","title":"When does return-conditioned supervised learning work for offline reinforcement learning?","arxiv_id":"2206.01079","date":"2022-06-02","proceeding":null,"authors":["David Brandfonbrener","Alberto Bietti","Jacob Buckman","Romain Laroche","Joan Bruna"],"abstract":"Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning on achieving high return. In this paper, we provide a rigorous study of the capabilities and limitations of RCSL, something which is crucially missing in previous work. We find that RCSL returns the optimal policy under a set of assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple point-mass environments and on datasets from the D4RL benchmark.","url_abs":"https://arxiv.org/abs/2206.01079v3","url_pdf":"https://arxiv.org/pdf/2206.01079v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-does-return-conditioned-supervised","repo_url":"https://github.com/davidbrandfonbrener/rcsl-paper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.01079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.01079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/davidbrandfonbrener/rcsl-paper","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0161b27cbe3cc58d","entry":"discount_cumsum","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"decision-transformer/gym/experiment.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/decision-transformer/gym/experiment.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0161b27cbe3cc58d"}},{"code_sha256_prefix":"fcd39de94f7cd1ea","entry":"collect_traj","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/jaxrl/rvs_evaluation.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/jaxrl/rvs_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcd39de94f7cd1ea"}},{"code_sha256_prefix":"facdc73f9b000275","entry":"default_init","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/jaxrl/networks/common.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/jaxrl/networks/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"facdc73f9b000275"}},{"code_sha256_prefix":"79152be87460ca81","entry":"evaluate","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/jaxrl/evaluation.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/jaxrl/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79152be87460ca81"}},{"code_sha256_prefix":"95424c4d978c1af3","entry":"evaluate","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/jaxrl/rvs_evaluation.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/jaxrl/rvs_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95424c4d978c1af3"}},{"code_sha256_prefix":"7aa880299f960110","entry":"generate_name","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"decision-transformer/gym/utils.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/decision-transformer/gym/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7aa880299f960110"}},{"code_sha256_prefix":"92258e3c17078ed0","entry":"generate_name","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/experiments/utils.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/experiments/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92258e3c17078ed0"}},{"code_sha256_prefix":"37e948f523448090","entry":"get_values","repo":"davidbrandfonbrener/rcsl-paper","repo_kind":"official","path":"jax_continuous_rl/jaxrl/networks/rvs_critics.py","file_url":"https://github.com/davidbrandfonbrener/rcsl-paper/blob/HEAD/jax_continuous_rl/jaxrl/networks/rvs_critics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37e948f523448090"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}