{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/curriculum-offline-imitation-learning","title":"Curriculum Offline Imitation Learning","arxiv_id":"2111.02056","date":"2021-11-03","proceeding":null,"authors":["Minghuan Liu","Hanye Zhao","Zhengyu Yang","Jian Shen","Weinan Zhang","Li Zhao","Tie-Yan Liu"],"abstract":"Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose \\textit{Curriculum Offline Imitation Learning (COIL)}, which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.","url_abs":"https://arxiv.org/abs/2111.02056v2","url_pdf":"https://arxiv.org/pdf/2111.02056v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"curriculum-offline-imitation-learning","repo_url":"https://github.com/apexrl/coil","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.02056","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.02056"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/aviralkumar2907/CQL","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lanyavik/BAIL","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/xbpeng/awr","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apexrl/coil","reach":null}],"summary":{"ran":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"37c462910b745910","entry":"COIL","repo":"apexrl/coil","repo_kind":"official","path":"rlkit/torch/coil/coil.py","file_url":"https://github.com/apexrl/coil/blob/HEAD/rlkit/torch/coil/coil.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37c462910b745910"}},{"code_sha256_prefix":"c03dde4a55ed93b3","entry":"get_numpy","repo":"apexrl/coil","repo_kind":"official","path":"rlkit/torch/coil/coil.py","file_url":"https://github.com/apexrl/coil/blob/HEAD/rlkit/torch/coil/coil.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c03dde4a55ed93b3"}},{"code_sha256_prefix":"711336253996e363","entry":"Trainer","repo":"apexrl/coil","repo_kind":"official","path":"rlkit/torch/coil/coil.py","file_url":"https://github.com/apexrl/coil/blob/HEAD/rlkit/torch/coil/coil.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"711336253996e363"}},{"code_sha256_prefix":"e98f92e9afd0ddda","entry":"tensor","repo":"apexrl/coil","repo_kind":"official","path":"rlkit/torch/coil/coil.py","file_url":"https://github.com/apexrl/coil/blob/HEAD/rlkit/torch/coil/coil.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e98f92e9afd0ddda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}