{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-in-distribution-success-scaling-curves","title":"Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalization","arxiv_id":"2502.18273","date":"2025-02-25","proceeding":null,"authors":["Ru Wang","Wei Huang","Selena Song","Haoyu Zhang","Yusuke Iwasawa","Yutaka Matsuo","Jiaxian Guo"],"abstract":"Generalization to novel compound tasks under distribution shift is important for deploying transformer-based language models (LMs). This work investigates Chain-of-Thought (CoT) reasoning as a means to enhance OOD generalization. Through controlled experiments across several compound tasks, we reveal three key insights: (1) While QA-trained models achieve near-perfect in-distribution accuracy, their OOD performance degrades catastrophically, even with 10000k+ training examples; (2) the granularity of CoT data strongly correlates with generalization performance; finer-grained CoT data leads to better generalization; (3) CoT exhibits remarkable sample efficiency, matching QA performance with much less (even 80%) data. Theoretically, we demonstrate that compound tasks inherently permit shortcuts in Q-A data that misalign with true reasoning principles, while CoT forces internalization of valid dependency structures, and thus can achieve better generalization. Further, we show that transformer positional embeddings can amplify generalization by emphasizing subtask condition recurrence in long CoT sequences. Our combined theoretical and empirical analysis provides compelling evidence for CoT reasoning as a crucial training paradigm for enabling LM generalization under real-world distributional shifts for compound tasks.","url_abs":"https://arxiv.org/abs/2502.18273v1","url_pdf":"https://arxiv.org/pdf/2502.18273v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-in-distribution-success-scaling-curves","repo_url":"https://github.com/physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.18273","atlas_url":"https://app.syntology.ai/?focus=2502.18273","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.18273"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization","reach":null}],"summary":{"ran_fixture":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"e6181fa93b6e7103","entry":"solve","repo":"physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization","repo_kind":"official","path":"LIS/data.py","file_url":"https://github.com/physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization/blob/HEAD/LIS/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6181fa93b6e7103"}},{"code_sha256_prefix":"2fd8910afd3ff7b7","entry":"solve_with_steps_accumulate_recap","repo":"physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization","repo_kind":"official","path":"LIS/data.py","file_url":"https://github.com/physicsru/Scaling-Curves-of-CoT-Granularity-for-Language-Model-Generalization/blob/HEAD/LIS/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fd8910afd3ff7b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}