{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/s1-simple-test-time-scaling","title":"s1: Simple test-time scaling","arxiv_id":"2501.19393","date":"2025-01-31","proceeding":null,"authors":["Niklas Muennighoff","Zitong Yang","Weijia Shi","Xiang Lisa Li","Li Fei-Fei","Hannaneh Hajishirzi","Luke Zettlemoyer","Percy Liang","Emmanuel Candès","Tatsunori Hashimoto"],"abstract":"Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending \"Wait\" multiple times to the model's generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1-32B with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/s1","url_abs":"https://arxiv.org/abs/2501.19393v2","url_pdf":"https://arxiv.org/pdf/2501.19393v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"s1-simple-test-time-scaling","repo_url":"https://github.com/simplescaling/s1","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"s1-simple-test-time-scaling","repo_url":"https://github.com/huggingface/open-r1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/mathematical-reasoning-on-aime24","task":"Mathematical Reasoning","dataset":"AIME24","model":"s1-32B","rank_in_archive_order":6,"of":9,"metrics":{"Acc":"56.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.19393","atlas_url":"https://app.syntology.ai/?focus=2501.19393","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.19393"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/vllm-project/vllm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/open-r1","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/simplescaling/s1","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00f5e7d9ee6636ba","entry":"get_sample_size","repo":"simplescaling/s1","repo_kind":"official","path":"eval/lm-evaluation-harness/lm_eval/evaluator_utils.py","file_url":"https://github.com/simplescaling/s1/blob/HEAD/eval/lm-evaluation-harness/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00f5e7d9ee6636ba"}},{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"simplescaling/s1","repo_kind":"official","path":"eval/lm-evaluation-harness/lm_eval/api/model.py","file_url":"https://github.com/simplescaling/s1/blob/HEAD/eval/lm-evaluation-harness/lm_eval/api/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256_prefix":"400ea4f64ccce5ef","entry":"parse_eval_args","repo":"simplescaling/s1","repo_kind":"official","path":"eval/lm-evaluation-harness/lm_eval/__main__.py","file_url":"https://github.com/simplescaling/s1/blob/HEAD/eval/lm-evaluation-harness/lm_eval/__main__.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"400ea4f64ccce5ef"}},{"code_sha256_prefix":"c44830a7722ff5f7","entry":"request_caching_arg_to_dict","repo":"simplescaling/s1","repo_kind":"official","path":"eval/lm-evaluation-harness/lm_eval/evaluator.py","file_url":"https://github.com/simplescaling/s1/blob/HEAD/eval/lm-evaluation-harness/lm_eval/evaluator.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c44830a7722ff5f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}