{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/should-robots-be-obedient","title":"Should Robots be Obedient?","arxiv_id":"1705.09990","date":"2017-05-28","proceeding":null,"authors":["Smitha Milli","Dylan Hadfield-Menell","Anca Dragan","Stuart Russell"],"abstract":"Intuitively, obedience -- following the order that a human gives -- seems\nlike a good property for a robot to have. But, we humans are not perfect and we\nmay give orders that are not best aligned to our preferences. We show that when\na human is not perfectly rational then a robot that tries to infer and act\naccording to the human's underlying preferences can always perform better than\na robot that simply follows the human's literal order. Thus, there is a\ntradeoff between the obedience of a robot and the value it can attain for its\nowner. We investigate how this tradeoff is impacted by the way the robot infers\nthe human's preferences, showing that some methods err more on the side of\nobedience than others. We then analyze how performance degrades when the robot\nhas a misspecified model of the features that the human cares about or the\nlevel of rationality of the human. Finally, we study how robots can start\ndetecting such model misspecification. Overall, our work suggests that there\nmight be a middle ground in which robots intelligently decide when to obey\nhuman orders, but err on the side of obedience.","url_abs":"http://arxiv.org/abs/1705.09990v1","url_pdf":"http://arxiv.org/pdf/1705.09990v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"should-robots-be-obedient","repo_url":"https://github.com/smilli/obedience","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1705.09990","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}