{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ai-safety-via-debate","title":"AI safety via debate","arxiv_id":"1805.00899","date":"2018-05-02","proceeding":null,"authors":["Geoffrey Irving","Paul Christiano","Dario Amodei"],"abstract":"To make AI systems broadly useful for challenging real-world tasks, we need\nthem to learn complex human goals and preferences. One approach to specifying\ncomplex goals asks humans to judge during training which agent behaviors are\nsafe and useful, but this approach can fail if the task is too complicated for\na human to directly judge. To help address this concern, we propose training\nagents via self play on a zero sum debate game. Given a question or proposed\naction, two agents take turns making short statements up to a limit, then a\nhuman judges which of the agents gave the most true, useful information. In an\nanalogy to complexity theory, debate with optimal play can answer any question\nin PSPACE given polynomial time judges (direct judging answers only NP\nquestions). In practice, whether debate works involves empirical questions\nabout humans and the tasks we want AIs to perform, plus theoretical questions\nabout the meaning of AI alignment. We report results on an initial MNIST\nexperiment where agents compete to convince a sparse classifier, boosting the\nclassifier's accuracy from 59.4% to 88.9% given 6 pixels and from 48.2% to\n85.2% given 4 pixels. Finally, we discuss theoretical and practical aspects of\nthe debate model, focusing on potential weaknesses as the model scales up, and\nwe propose future human and computer experiments to test these properties.","url_abs":"http://arxiv.org/abs/1805.00899v2","url_pdf":"http://arxiv.org/pdf/1805.00899v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ai-safety-via-debate","repo_url":"https://github.com/google-deepmind/debate","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"ai-safety-via-debate","repo_url":"https://github.com/jvmancuso/safe-debates","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.00899","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.00899"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/debate","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jvmancuso/safe-debates","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a147d99607177105","entry":"prepare_data","repo":"jvmancuso/safe-debates","repo_kind":"listed","path":"prepare_data.py","file_url":"https://github.com/jvmancuso/safe-debates/blob/HEAD/prepare_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a147d99607177105"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}