{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/let-s-verify-step-by-step-1","title":"Let's Verify Step by Step","arxiv_id":"2305.20050","date":"2023-05-31","proceeding":"Preprint 2023 5","authors":["Hunter Lightman","Vineet Kosaraju","Yura Burda","Harri Edwards","Bowen Baker","Teddy Lee","Jan Leike","John Schulman","Ilya Sutskever","Karl Cobbe"],"abstract":"In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.","url_abs":"https://arxiv.org/abs/2305.20050v1","url_pdf":"https://arxiv.org/pdf/2305.20050v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"let-s-verify-step-by-step-1","repo_url":"https://github.com/openai/prm800k","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"let-s-verify-step-by-step-1","repo_url":"https://github.com/consequentai/fneval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"let-s-verify-step-by-step-1","repo_url":"https://github.com/gentopia-ai/gentopia","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"math","task_name":"Math"},{"task_slug":"math-word-problem-solving","task_name":"Math Word Problem Solving"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[{"slug":"prm800k","name":"PRM800K","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/math-word-problem-solving-on-math-minival","task":"Math Word Problem Solving","dataset":"MATH minival","model":"Process Supervision (GPT-4)","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"78.2"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2305.20050","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.20050"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/consequentai/fneval","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/prm800k","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gentopia-ai/gentopia","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bc7ab5cea18531ef","entry":"are_equal_under_sympy","repo":"openai/prm800k","repo_kind":"official","path":"prm800k/grading/grader.py","file_url":"https://github.com/openai/prm800k/blob/HEAD/prm800k/grading/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc7ab5cea18531ef"}},{"code_sha256_prefix":"368d3db6347269c3","entry":"count_unknown_letters_in_expr","repo":"openai/prm800k","repo_kind":"official","path":"prm800k/grading/grader.py","file_url":"https://github.com/openai/prm800k/blob/HEAD/prm800k/grading/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"368d3db6347269c3"}},{"code_sha256_prefix":"95f02454eddf082a","entry":"should_allow_eval","repo":"openai/prm800k","repo_kind":"official","path":"prm800k/grading/grader.py","file_url":"https://github.com/openai/prm800k/blob/HEAD/prm800k/grading/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95f02454eddf082a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}