{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/concrete-problems-in-ai-safety","title":"Concrete Problems in AI Safety","arxiv_id":"1606.06565","date":"2016-06-21","proceeding":null,"authors":["Dario Amodei","Chris Olah","Jacob Steinhardt","Paul Christiano","John Schulman","Dan Mané"],"abstract":"Rapid progress in machine learning and artificial intelligence (AI) has\nbrought increasing attention to the potential impacts of AI technologies on\nsociety. In this paper we discuss one such potential impact: the problem of\naccidents in machine learning systems, defined as unintended and harmful\nbehavior that may emerge from poor design of real-world AI systems. We present\na list of five practical research problems related to accident risk,\ncategorized according to whether the problem originates from having the wrong\nobjective function (\"avoiding side effects\" and \"avoiding reward hacking\"), an\nobjective function that is too expensive to evaluate frequently (\"scalable\nsupervision\"), or undesirable behavior during the learning process (\"safe\nexploration\" and \"distributional shift\"). We review previous work in these\nareas as well as suggesting research directions with a focus on relevance to\ncutting-edge AI systems. Finally, we consider the high-level question of how to\nthink most productively about the safety of forward-looking applications of AI.","url_abs":"http://arxiv.org/abs/1606.06565v2","url_pdf":"http://arxiv.org/pdf/1606.06565v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"concrete-problems-in-ai-safety","repo_url":"https://github.com/mateuszjurewicz/bornhack_ml_crashcourse","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"safe-exploration","task_name":"Safe Exploration"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1606.06565","atlas_url":"https://app.syntology.ai/?focus=1606.06565","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}