{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semi-supervised-offline-reinforcement-1","title":"Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories","arxiv_id":"2210.06518","date":"2022-10-12","proceeding":null,"authors":["Qinqing Zheng","Mikael Henaff","Brandon Amos","Aditya Grover"],"abstract":"Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements. We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting. Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action and reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information. For this setting, we develop and study a simple meta-algorithmic pipeline that learns an inverse dynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories. Empirically, we find this simple pipeline to be highly successful -- on several D4RL benchmarks~\\cite{fu2020d4rl}, certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10\\% of trajectories which are highly suboptimal. To strengthen our understanding, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e.g., choice of inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.","url_abs":"https://arxiv.org/abs/2210.06518v3","url_pdf":"https://arxiv.org/pdf/2210.06518v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semi-supervised-offline-reinforcement-1","repo_url":"https://github.com/facebookresearch/ssorl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.06518","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06518"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sfujim/TD3_BC","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/ssorl","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/online-dt","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/scottemmons/youngs-cql","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":2,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20922c79a0929bbf","entry":"flatten","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"viskit/core.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20922c79a0929bbf"}},{"code_sha256_prefix":"32231080b4b40efc","entry":"batch_to_torch","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"SimpleSAC/replay_buffer.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/SimpleSAC/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32231080b4b40efc"}},{"code_sha256_prefix":"1e5f5f635bdf773a","entry":"extend_and_repeat","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"SimpleSAC/model.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/SimpleSAC/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e5f5f635bdf773a"}},{"code_sha256_prefix":"965e75a1245dd833","entry":"load_progress","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"viskit/core.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"965e75a1245dd833"}},{"code_sha256_prefix":"6c7a4a9349a395e9","entry":"unique","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"viskit/core.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c7a4a9349a395e9"}},{"code_sha256_prefix":"bf34e2eb1af0b2ba","entry":"concatenate_batches","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"SimpleSAC/replay_buffer.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/SimpleSAC/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf34e2eb1af0b2ba"}},{"code_sha256_prefix":"5d7d29c4e6cd447f","entry":"subsample_batch","repo":"scottemmons/youngs-cql","repo_kind":"found_in_text","path":"SimpleSAC/replay_buffer.py","file_url":"https://github.com/scottemmons/youngs-cql/blob/HEAD/SimpleSAC/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d7d29c4e6cd447f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}