{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-efficient-hierarchical-reinforcement","title":"Data-Efficient Hierarchical Reinforcement Learning","arxiv_id":"1805.08296","date":"2018-05-21","proceeding":"NeurIPS 2018 12","authors":["Ofir Nachum","Shixiang Gu","Honglak Lee","Sergey Levine"],"abstract":"Hierarchical reinforcement learning (HRL) is a promising approach to extend\ntraditional reinforcement learning (RL) methods to solve more complex tasks.\nYet, the majority of current HRL methods require careful task-specific design\nand on-policy training, making them difficult to apply in real-world scenarios.\nIn this paper, we study how we can develop HRL algorithms that are general, in\nthat they do not make onerous additional assumptions beyond standard RL\nalgorithms, and efficient, in the sense that they can be used with modest\nnumbers of interaction samples, making them suitable for real-world problems\nsuch as robotic control. For generality, we develop a scheme where lower-level\ncontrollers are supervised with goals that are learned and proposed\nautomatically by the higher-level controllers. To address efficiency, we\npropose to use off-policy experience for both higher and lower-level training.\nThis poses a considerable challenge, since changes to the lower-level behaviors\nchange the action space for the higher-level policy, and we introduce an\noff-policy correction to remedy this challenge. This allows us to take\nadvantage of recent advances in off-policy model-free RL to learn both higher-\nand lower-level policies using substantially fewer environment interactions\nthan on-policy algorithms. We term the resulting HRL agent HIRO and find that\nit is generally applicable and highly sample-efficient. Our experiments show\nthat HIRO can be used to learn highly complex behaviors for simulated robots,\nsuch as pushing objects and utilizing them to reach target locations, learning\nfrom only a few million samples, equivalent to a few days of real-time\ninteraction. In comparisons with a number of prior HRL methods, we find that\nour approach substantially outperforms previous state-of-the-art techniques.","url_abs":"http://arxiv.org/abs/1805.08296v4","url_pdf":"http://arxiv.org/pdf/1805.08296v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/tensorflow/models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/AlexZhaoZt/Temporal_Leap_HRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/brandontrabucco/efficient-hrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/cosynus-lix/STAR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/hebowei2000/deep-reinforcement-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/mindspore-courses/Deep-Reinforcement-Learning-Algorithms-with-MindSpore","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/sumkumar/hiro_impl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/tensorflow/models/tree/master/research/efficient-hrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/watakandai/hiro_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/ziangqin-stu/impl_data-effiient-hrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/ziangqin-stu/impl_hiro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"data-efficient-hierarchical-reinforcement","repo_url":"https://github.com/ziangqin-stu/rl_hiro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"hierarchical-reinforcement-learning","task_name":"Hierarchical Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.08296","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}