{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-relationship-on-learning-mathematical","title":"Scaling Relationship on Learning Mathematical Reasoning with Large Language Models","arxiv_id":"2308.01825","date":"2023-08-03","proceeding":null,"authors":["Zheng Yuan","Hongyi Yuan","Chengpeng Li","Guanting Dong","Keming Lu","Chuanqi Tan","Chang Zhou","Jingren Zhou"],"abstract":"Mathematical reasoning is a challenging task for large language models (LLMs), while the scaling relationship of it with respect to LLM capacity is under-explored. In this paper, we investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that pre-training loss is a better indicator of the model's performance than the model's parameter count. We apply supervised fine-tuning (SFT) with different amounts of supervised data and empirically find a log-linear relation between data amount and model performance, and we find better models improve less with enlarged supervised datasets. To augment more data samples for improving model performances without any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT uses supervised models to generate and collect correct reasoning paths as augmented fine-tuning datasets. We find with augmented samples containing more distinct reasoning paths, RFT improves mathematical reasoning performance more for LLMs. We also find RFT brings more improvement for less performant LLMs. Furthermore, we combine rejection samples from multiple models which push LLaMA-7B to an accuracy of 49.3\\% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9\\% significantly.","url_abs":"https://arxiv.org/abs/2308.01825v2","url_pdf":"https://arxiv.org/pdf/2308.01825v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-relationship-on-learning-mathematical","repo_url":"https://github.com/ofa-sys/gsm8k-screl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"RFT 70B","rank_in_archive_order":111,"of":164,"metrics":{"Accuracy":"64.8","Parameters (Billion)":"79"},"uses_additional_data":true},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"RFT 13B","rank_in_archive_order":126,"of":164,"metrics":{"Accuracy":"55.3","Parameters (Billion)":"13"},"uses_additional_data":true},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"RFT 7B","rank_in_archive_order":134,"of":164,"metrics":{"Accuracy":"51.2","Parameters (Billion)":"7"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.01825","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.01825"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ofa-sys/gsm8k-screl","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4674cf2055f705c7","entry":"check_equation","repo":"ofa-sys/gsm8k-screl","repo_kind":"official","path":"collect_rejection_sampling.py","file_url":"https://github.com/ofa-sys/gsm8k-screl/blob/HEAD/collect_rejection_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4674cf2055f705c7"}},{"code_sha256_prefix":"e94e204600c3de88","entry":"extract_answer","repo":"ofa-sys/gsm8k-screl","repo_kind":"official","path":"collect_rejection_sampling.py","file_url":"https://github.com/ofa-sys/gsm8k-screl/blob/HEAD/collect_rejection_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e94e204600c3de88"}},{"code_sha256_prefix":"5ae6f5f7da3c0933","entry":"parse_gold","repo":"ofa-sys/gsm8k-screl","repo_kind":"official","path":"collect_rejection_sampling.py","file_url":"https://github.com/ofa-sys/gsm8k-screl/blob/HEAD/collect_rejection_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ae6f5f7da3c0933"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}