{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/composable-deep-reinforcement-learning-for","title":"Composable Deep Reinforcement Learning for Robotic Manipulation","arxiv_id":"1803.06773","date":"2018-03-19","proceeding":null,"authors":["Tuomas Haarnoja","Vitchyr Pong","Aurick Zhou","Murtaza Dalal","Pieter Abbeel","Sergey Levine"],"abstract":"Model-free deep reinforcement learning has been shown to exhibit good\nperformance in domains ranging from video games to simulated robotic\nmanipulation and locomotion. However, model-free methods are known to perform\npoorly when the interaction time with the environment is limited, as is the\ncase for most real-world robotic tasks. In this paper, we study how maximum\nentropy policies trained using soft Q-learning can be applied to real-world\nrobotic manipulation. The application of this method to real-world manipulation\nis facilitated by two important features of soft Q-learning. First, soft\nQ-learning can learn multimodal exploration strategies by learning policies\nrepresented by expressive energy-based models. Second, we show that policies\nlearned with soft Q-learning can be composed to create new policies, and that\nthe optimality of the resulting policy can be bounded in terms of the\ndivergence between the composed policies. This compositionality provides an\nespecially valuable tool for real-world manipulation, where constructing new\npolicies by composing existing skills can provide a large gain in efficiency\nover training from scratch. Our experimental evaluation demonstrates that soft\nQ-learning is substantially more sample efficient than prior model-free deep\nreinforcement learning methods, and that compositionality can be performed for\nboth simulated and real-world tasks.","url_abs":"http://arxiv.org/abs/1803.06773v1","url_pdf":"http://arxiv.org/pdf/1803.06773v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"composable-deep-reinforcement-learning-for","repo_url":"https://github.com/haarnoja/softqlearning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.06773","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}