{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystifying-long-chain-of-thought-reasoning","title":"Demystifying Long Chain-of-Thought Reasoning in LLMs","arxiv_id":"2502.03373","date":"2025-02-05","proceeding":null,"authors":["Edward Yeo","Yuxuan Tong","Morry Niu","Graham Neubig","Xiang Yue"],"abstract":"Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the mechanics of long CoT reasoning, identifying the key factors that enable models to generate long CoT trajectories. Through extensive supervised fine-tuning (SFT) and RL experiments, we present four main findings: (1) While SFT is not strictly necessary, it simplifies training and improves efficiency; (2) Reasoning capabilities tend to emerge with increased training compute, but their development is not guaranteed, making reward shaping crucial for stabilizing CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We find that leveraging noisy, web-extracted solutions with filtering mechanisms shows strong potential, particularly for out-of-distribution (OOD) tasks such as STEM reasoning; and (4) Core abilities like error correction are inherently present in base models, but incentivizing these skills effectively for complex tasks via RL demands significant compute, and measuring their emergence requires a nuanced approach. These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Our code is available at: https://github.com/eddycmu/demystify-long-cot.","url_abs":"https://arxiv.org/abs/2502.03373v1","url_pdf":"https://arxiv.org/pdf/2502.03373v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystifying-long-chain-of-thought-reasoning","repo_url":"https://github.com/eddycmu/demystify-long-cot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.03373","atlas_url":"https://app.syntology.ai/?focus=2502.03373","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.03373"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eddycmu/demystify-long-cot","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b6035617946dd129","entry":"compute_approx_kl","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/models/utils.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6035617946dd129"}},{"code_sha256_prefix":"1db73767f2ebd35e","entry":"compute_reward","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/models/utils.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1db73767f2ebd35e"}},{"code_sha256_prefix":"5be0d1b8e71bbd1e","entry":"get_shingles","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"minhash/find_similar_strings.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/minhash/find_similar_strings.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5be0d1b8e71bbd1e"}},{"code_sha256_prefix":"959df7824bf09cc1","entry":"log_probs_from_logits","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/models/utils.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"959df7824bf09cc1"}},{"code_sha256_prefix":"edc77f8288a52828","entry":"sanitize_filename","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"minhash/find_similar_strings.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/minhash/find_similar_strings.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edc77f8288a52828"}},{"code_sha256_prefix":"d66f259aaf6cd9c7","entry":"convert_ring_attn_params","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/models/ring_attn_utils.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/models/ring_attn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d66f259aaf6cd9c7"}},{"code_sha256_prefix":"e93117d99aa6b210","entry":"reset_ring_attn_position_ids","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/models/ring_attn_utils.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/models/ring_attn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e93117d99aa6b210"}},{"code_sha256_prefix":"ca3514c46a1261d7","entry":"strip_sequence","repo":"eddycmu/demystify-long-cot","repo_kind":"official","path":"openrlhf/openrlhf/cli/serve_rm.py","file_url":"https://github.com/eddycmu/demystify-long-cot/blob/HEAD/openrlhf/openrlhf/cli/serve_rm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca3514c46a1261d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}