{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/at-which-training-stage-does-cocde-data-help","title":"At Which Training Stage Does Code Data Help LLMs Reasoning?","arxiv_id":"2309.16298","date":"2023-09-28","proceeding":null,"authors":["Yingwei Ma","Yue Liu","Yue Yu","Yuanliang Zhang","Yu Jiang","Changjian Wang","Shanshan Li"],"abstract":"Large Language Models (LLMs) have exhibited remarkable reasoning capabilities and become the foundation of language technologies. Inspired by the great success of code data in training LLMs, we naturally wonder at which training stage introducing code data can really help LLMs reasoning. To this end, this paper systematically explores the impact of code data on LLMs at different stages. Concretely, we introduce the code data at the pre-training stage, instruction-tuning stage, and both of them, respectively. Then, the reasoning capability of LLMs is comprehensively and fairly evaluated via six reasoning tasks in five domains. We critically analyze the experimental results and provide conclusions with insights. First, pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. Besides, at the instruction-tuning stage, code data endows LLMs the task-specific reasoning capability. Moreover, the dynamic mixing strategy of code and text data assists LLMs to learn reasoning capability step-by-step during training. These insights deepen the understanding of LLMs regarding reasoning ability for their application, such as scientific question answering, legal support, etc. The source code and model parameters are released at the link:~\\url{https://github.com/yingweima2022/CodeLLM}.","url_abs":"https://arxiv.org/abs/2309.16298v2","url_pdf":"https://arxiv.org/pdf/2309.16298v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"at-which-training-stage-does-cocde-data-help","repo_url":"https://github.com/yingweima2022/codellm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.16298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yingweima2022/codellm","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ba31aa28df517100","entry":"convert_text_to_ids","repo":"yingweima2022/codellm","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/src/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba31aa28df517100"}},{"code_sha256_prefix":"2da848988b60d9cc","entry":"divideIntoNstrand","repo":"yingweima2022/codellm","repo_kind":"official","path":"process_multi_level.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/process_multi_level.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2da848988b60d9cc"}},{"code_sha256_prefix":"a43454e73b20e26c","entry":"gather","repo":"yingweima2022/codellm","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/src/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a43454e73b20e26c"}},{"code_sha256_prefix":"f6086c3d6ab0cde0","entry":"get_input_data_batch_slice_map","repo":"yingweima2022/codellm","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6086c3d6ab0cde0"}},{"code_sha256_prefix":"0e9b480c03a3d45b","entry":"get_input_data_batch_slice_map1","repo":"yingweima2022/codellm","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e9b480c03a3d45b"}},{"code_sha256_prefix":"c5e39df8dfefe276","entry":"get_jsonl_files","repo":"yingweima2022/codellm","repo_kind":"official","path":"process_multi_level.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/process_multi_level.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5e39df8dfefe276"}},{"code_sha256_prefix":"f12f9486ba2960e6","entry":"set_weight_decay","repo":"yingweima2022/codellm","repo_kind":"official","path":"instruction_following.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/instruction_following.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f12f9486ba2960e6"}},{"code_sha256_prefix":"9b45cbca36a0a377","entry":"compute_loss","repo":"yingweima2022/codellm","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/yingweima2022/codellm/blob/HEAD/src/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b45cbca36a0a377"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}