{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalable-agent-alignment-via-reward-modeling","title":"Scalable agent alignment via reward modeling: a research direction","arxiv_id":"1811.07871","date":"2018-11-19","proceeding":null,"authors":["Jan Leike","David Krueger","Tom Everitt","Miljan Martic","Vishal Maini","Shane Legg"],"abstract":"One obstacle to applying reinforcement learning algorithms to real-world\nproblems is the lack of suitable reward functions. Designing such reward\nfunctions is difficult in part because the user only has an implicit\nunderstanding of the task objective. This gives rise to the agent alignment\nproblem: how do we create agents that behave in accordance with the user's\nintentions? We outline a high-level research direction to solve the agent\nalignment problem centered around reward modeling: learning a reward function\nfrom interaction with the user and optimizing the learned reward function with\nreinforcement learning. We discuss the key challenges we expect to face when\nscaling reward modeling to complex and general domains, concrete approaches to\nmitigate these challenges, and ways to establish trust in the resulting agents.","url_abs":"http://arxiv.org/abs/1811.07871v1","url_pdf":"http://arxiv.org/pdf/1811.07871v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scalable-agent-alignment-via-reward-modeling","repo_url":"https://github.com/Angeliki-Panoulia/CallForCode","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"scalable-agent-alignment-via-reward-modeling","repo_url":"https://github.com/rddy/ReQueST","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"scalable-agent-alignment-via-reward-modeling","repo_url":"https://github.com/rgreenblatt/ai_alignment_readings","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1811.07871","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}