{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-ensemble-trust-region-policy","title":"Model-Ensemble Trust-Region Policy Optimization","arxiv_id":"1802.10592","date":"2018-02-28","proceeding":"ICLR 2018 1","authors":["Thanard Kurutach","Ignasi Clavera","Yan Duan","Aviv Tamar","Pieter Abbeel"],"abstract":"Model-free reinforcement learning (RL) methods are succeeding in a growing\nnumber of tasks, aided by recent advances in deep learning. However, they tend\nto suffer from high sample complexity, which hinders their use in real-world\ndomains. Alternatively, model-based reinforcement learning promises to reduce\nsample complexity, but tends to require careful tuning and to date have\nsucceeded mainly in restrictive domains where simple models are sufficient for\nlearning. In this paper, we analyze the behavior of vanilla model-based\nreinforcement learning methods when deep neural networks are used to learn both\nthe model and the policy, and show that the learned policy tends to exploit\nregions where insufficient data is available for the model to be learned,\ncausing instability in training. To overcome this issue, we propose to use an\nensemble of models to maintain the model uncertainty and regularize the\nlearning process. We further show that the use of likelihood ratio derivatives\nyields much more stable learning than backpropagation through time. Altogether,\nour approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO)\nsignificantly reduces the sample complexity compared to model-free deep RL\nmethods on challenging continuous control benchmark tasks.","url_abs":"http://arxiv.org/abs/1802.10592v2","url_pdf":"http://arxiv.org/pdf/1802.10592v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-ensemble-trust-region-policy","repo_url":"https://github.com/thanard/me-trpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"model-ensemble-trust-region-policy","repo_url":"https://github.com/WilsonWangTHU/mbbl-metrpo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"model-based-reinforcement-learning","task_name":"Model-based Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1802.10592","atlas_url":"https://app.syntology.ai/?focus=1802.10592","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.10592"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/WilsonWangTHU/mbbl-metrpo","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thanard/me-trpo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e9d3121fd4274709","entry":"build_dynamics_graph","repo":"thanard/me-trpo","repo_kind":"official","path":"model_based_rl.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/model_based_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9d3121fd4274709"}},{"code_sha256_prefix":"adb6b42c4e28a0b8","entry":"build_policy_graph","repo":"thanard/me-trpo","repo_kind":"official","path":"model_based_rl.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/model_based_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adb6b42c4e28a0b8"}},{"code_sha256_prefix":"1889cae48084f14c","entry":"get_action","repo":"thanard/me-trpo","repo_kind":"official","path":"env_helpers.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/env_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1889cae48084f14c"}},{"code_sha256_prefix":"51d17f38112bbd30","entry":"get_dynamics_optimizer","repo":"thanard/me-trpo","repo_kind":"official","path":"model_based_rl.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/model_based_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51d17f38112bbd30"}},{"code_sha256_prefix":"9cad5fcb225efa45","entry":"prepare_policy","repo":"thanard/me-trpo","repo_kind":"official","path":"env_helpers.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/env_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cad5fcb225efa45"}},{"code_sha256_prefix":"df94ecc6a7d6bd3a","entry":"svg_gradient","repo":"thanard/me-trpo","repo_kind":"official","path":"svg_utils.py","file_url":"https://github.com/thanard/me-trpo/blob/HEAD/svg_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df94ecc6a7d6bd3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}