{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/imitate-explore-and-self-improve-a","title":"Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems","arxiv_id":"2412.09413","date":"2024-12-12","proceeding":null,"authors":["Yingqian Min","Zhipeng Chen","Jinhao Jiang","Jie Chen","Jia Deng","Yiwen Hu","Yiru Tang","Jiapeng Wang","Xiaoxue Cheng","Huatong Song","Wayne Xin Zhao","Zheng Liu","Zhongyuan Wang","Ji-Rong Wen"],"abstract":"Recently, slow-thinking reasoning systems, such as o1, have demonstrated remarkable capabilities in solving complex reasoning tasks. These systems typically engage in an extended thinking process before responding to a query, allowing them to generate more thorough, accurate, and well-reasoned solutions. These systems are primarily developed and maintained by industry, with their core techniques not publicly disclosed. In response, an increasing number of studies from the research community aim to explore the technical foundations underlying these powerful reasoning systems. Building on these prior efforts, this paper presents a reproduction report on implementing o1-like reasoning systems. We introduce an ``imitate, explore, and self-improve'' framework, denoted as \\textbf{STILL-2}, as our primary technical approach to train the reasoning model. In the initial phase, we use distilled long-form thought data to fine-tune the reasoning model, enabling it to invoke a slow-thinking mode. The model is then encouraged to explore challenging problems by generating multiple rollouts, which can result in increasingly more high-quality trajectories that lead to correct answers. Furthermore, the model undergoes self-improvement by iteratively refining its training dataset. To verify the effectiveness of this approach, we conduct extensive experiments on three challenging benchmarks. The experimental results demonstrate that our approach achieves competitive performance compared to industry-level reasoning systems on these benchmarks.","url_abs":"https://arxiv.org/abs/2412.09413v2","url_pdf":"https://arxiv.org/pdf/2412.09413v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"imitate-explore-and-self-improve-a","repo_url":"https://github.com/RUCAIBox/Slow_Thinking_with_LLMs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"imitate-explore-and-self-improve-a","repo_url":"https://github.com/novasky-ai/skythought","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"imitate-explore-and-self-improve-a","repo_url":"https://github.com/rucaibox/virgo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.09413","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.09413"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/novasky-ai/skythought","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/virgo","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RUCAIBox/Slow_Thinking_with_LLMs","reach":null}],"summary":{"ran_draft_wrong":4,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"a587cfb03e7e3adf","entry":"add_code_prefix","repo":"RUCAIBox/Slow_Thinking_with_LLMs","repo_kind":"official","path":"STILL-3-TOOL/data_synthesis/coding_by_thinking.py","file_url":"https://github.com/RUCAIBox/Slow_Thinking_with_LLMs/blob/HEAD/STILL-3-TOOL/data_synthesis/coding_by_thinking.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a587cfb03e7e3adf"}},{"code_sha256_prefix":"60aec79830c634c8","entry":"create_test_prompt","repo":"rucaibox/virgo","repo_kind":"listed","path":"evaluation/MathVerse/evaluation/extract_answer_s1.py","file_url":"https://github.com/rucaibox/virgo/blob/HEAD/evaluation/MathVerse/evaluation/extract_answer_s1.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60aec79830c634c8"}},{"code_sha256_prefix":"4b0aef908978da14","entry":"load_jsonl","repo":"RUCAIBox/Slow_Thinking_with_LLMs","repo_kind":"official","path":"STILL-3-TOOL/data_synthesis/coding_by_thinking.py","file_url":"https://github.com/RUCAIBox/Slow_Thinking_with_LLMs/blob/HEAD/STILL-3-TOOL/data_synthesis/coding_by_thinking.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b0aef908978da14"}},{"code_sha256_prefix":"8a715d0671890763","entry":"process_input","repo":"RUCAIBox/Slow_Thinking_with_LLMs","repo_kind":"official","path":"STILL-3-TOOL/data_synthesis/coding_by_thinking.py","file_url":"https://github.com/RUCAIBox/Slow_Thinking_with_LLMs/blob/HEAD/STILL-3-TOOL/data_synthesis/coding_by_thinking.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a715d0671890763"}},{"code_sha256_prefix":"350d0bc5afcc4922","entry":"verify_extraction","repo":"rucaibox/virgo","repo_kind":"listed","path":"evaluation/MathVerse/evaluation/extract_answer_s1.py","file_url":"https://github.com/rucaibox/virgo/blob/HEAD/evaluation/MathVerse/evaluation/extract_answer_s1.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"350d0bc5afcc4922"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}