{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reward-learning-from-human-preferences-and","title":"Reward learning from human preferences and demonstrations in Atari","arxiv_id":"1811.06521","date":"2018-11-15","proceeding":"NeurIPS 2018 12","authors":["Borja Ibarz","Jan Leike","Tobias Pohlen","Geoffrey Irving","Shane Legg","Dario Amodei"],"abstract":"To solve complex real-world problems with reinforcement learning, we cannot\nrely on manually specified reward functions. Instead, we can have humans\ncommunicate an objective to the agent directly. In this work, we combine two\napproaches to learning from human feedback: expert demonstrations and\ntrajectory preferences. We train a deep neural network to model the reward\nfunction and use its predicted reward to train an DQN-based deep reinforcement\nlearning agent on 9 Atari games. Our approach beats the imitation learning\nbaseline in 7 games and achieves strictly superhuman performance on 2 games\nwithout using game rewards. Additionally, we investigate the goodness of fit of\nthe reward model, present some reward hacking problems, and study the effects\nof noise in the human labels.","url_abs":"http://arxiv.org/abs/1811.06521v1","url_pdf":"http://arxiv.org/pdf/1811.06521v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reward-learning-from-human-preferences-and","repo_url":"https://github.com/asjad99/Reinforcement-Learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"reward-learning-from-human-preferences-and","repo_url":"https://github.com/rddy/ReQueST","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1811.06521","atlas_url":"https://app.syntology.ai/?focus=1811.06521","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}