{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/common-7b-language-models-already-possess","title":"Common 7B Language Models Already Possess Strong Math Capabilities","arxiv_id":"2403.04706","date":"2024-03-07","proceeding":null,"authors":["Chen Li","Weiqi Wang","Jingcheng Hu","Yixuan Wei","Nanning Zheng","Han Hu","Zheng Zhang","Houwen Peng"],"abstract":"Mathematical capabilities were previously believed to emerge in common language models only at a very large scale or require extensive math-related pre-training. This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities, as evidenced by its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks, respectively, when selecting the best response from 256 random generations. The primary issue with the current base model is the difficulty in consistently eliciting its inherent mathematical capabilities. Notably, the accuracy for the first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks, respectively. We find that simply scaling up the SFT data can significantly enhance the reliability of generating correct answers. However, the potential for extensive scaling is constrained by the scarcity of publicly available math questions. To overcome this limitation, we employ synthetic data, which proves to be nearly as effective as real data and shows no clear saturation when scaled up to approximately one million samples. This straightforward approach achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B models, surpassing previous models by 14.2% and 20.8%, respectively. We also provide insights into scaling behaviors across different reasoning complexities and error types.","url_abs":"https://arxiv.org/abs/2403.04706v1","url_pdf":"https://arxiv.org/pdf/2403.04706v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"common-7b-language-models-already-possess","repo_url":"https://github.com/xwin-lm/xwin-lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"common-7b-language-models-already-possess","repo_url":"https://github.com/jerrywu-code/susgen","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.04706","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.04706"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jerrywu-code/susgen","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xwin-lm/xwin-lm","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":5,"unverified":22},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1},"listed":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"7a733b18081436f3","entry":"extract_text","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/HumanEval/humaneval_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/HumanEval/humaneval_gen.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a733b18081436f3"}},{"code_sha256_prefix":"f05983c38acfa250","entry":"find_indentation","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/APPS/reindent.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/APPS/reindent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f05983c38acfa250"}},{"code_sha256_prefix":"2763cff93674bd85","entry":"generate_prompt","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/DS1000/ds1000_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/DS1000/ds1000_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2763cff93674bd85"}},{"code_sha256_prefix":"34de3c98b4c4ccf0","entry":"generate_prompt","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/HumanEval/humaneval_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/HumanEval/humaneval_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34de3c98b4c4ccf0"}},{"code_sha256_prefix":"4557da831202481d","entry":"import_source_file","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/DS1000/ds1000ds.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/DS1000/ds1000ds.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4557da831202481d"}},{"code_sha256_prefix":"27f23c533d62061a","entry":"replace_inline_tabs","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/APPS/reindent.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/APPS/reindent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27f23c533d62061a"}},{"code_sha256_prefix":"b26735c9048e647a","entry":"change_format","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/DS1000/ds1000_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/DS1000/ds1000_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b26735c9048e647a"}},{"code_sha256_prefix":"ce0b6e5c855875fc","entry":"documentMetadata_loader","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"utils/extractText_sus.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/utils/extractText_sus.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce0b6e5c855875fc"}},{"code_sha256_prefix":"30e951ff25cd61d7","entry":"extract_answer","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"eval/code/eval_finqa.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/eval/code/eval_finqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30e951ff25cd61d7"}},{"code_sha256_prefix":"72dfc5e9d73fa763","entry":"extract_numbers","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"eval/code/eval_finqa.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/eval/code/eval_finqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72dfc5e9d73fa763"}},{"code_sha256_prefix":"aa963276f597a210","entry":"generate_text","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa963276f597a210"}},{"code_sha256_prefix":"15a54731c6978e32","entry":"get_model","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/HumanEval/humaneval_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/HumanEval/humaneval_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15a54731c6978e32"}},{"code_sha256_prefix":"54ff8bf66412b359","entry":"get_model","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/MBPP/mbpp_gen.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/MBPP/mbpp_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54ff8bf66412b359"}},{"code_sha256_prefix":"ed8331339b4de97c","entry":"get_question","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/APPS/testing_util.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/APPS/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed8331339b4de97c"}},{"code_sha256_prefix":"0c83bb0f43b8dc57","entry":"get_solutions","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/APPS/testing_util.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/APPS/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c83bb0f43b8dc57"}},{"code_sha256_prefix":"8bbb265e59e23293","entry":"get_valid_problems","repo":"xwin-lm/xwin-lm","repo_kind":"official","path":"Xwin-Coder/APPS/testing_util.py","file_url":"https://github.com/xwin-lm/xwin-lm/blob/HEAD/Xwin-Coder/APPS/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bbb265e59e23293"}},{"code_sha256_prefix":"5adc840cfa3cba8a","entry":"inference","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/lora_inference.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/lora_inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5adc840cfa3cba8a"}},{"code_sha256_prefix":"cd3318f643f2b529","entry":"instr_prompt","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd3318f643f2b529"}},{"code_sha256_prefix":"836aab9d5828036c","entry":"llama3_short","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/utils/prompt_template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/utils/prompt_template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"836aab9d5828036c"}},{"code_sha256_prefix":"43fd4e86eac55a80","entry":"load_datasets","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"utils/loadhf_data.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/utils/loadhf_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43fd4e86eac55a80"}},{"code_sha256_prefix":"b3cd6beeda0e0909","entry":"load_json","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/finetune.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3cd6beeda0e0909"}},{"code_sha256_prefix":"991cad3f55659f76","entry":"load_jsonl","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"utils/process_v1.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/utils/process_v1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"991cad3f55659f76"}},{"code_sha256_prefix":"5acbb99c75b740a1","entry":"load_lora","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/lora_inference.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/lora_inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5acbb99c75b740a1"}},{"code_sha256_prefix":"c87d1b4488a54c82","entry":"load_model","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c87d1b4488a54c82"}},{"code_sha256_prefix":"c09691f510dbbb2a","entry":"mistral_formal","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/utils/prompt_template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/utils/prompt_template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c09691f510dbbb2a"}},{"code_sha256_prefix":"46788756dee21d65","entry":"mistral_short","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/utils/prompt_template.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/utils/prompt_template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46788756dee21d65"}},{"code_sha256_prefix":"1efc261c7054e106","entry":"time_logger","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/batch_inference.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/batch_inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1efc261c7054e106"}},{"code_sha256_prefix":"448dfa384a102a35","entry":"tokenize","repo":"jerrywu-code/susgen","repo_kind":"listed","path":"src/finetune.py","file_url":"https://github.com/jerrywu-code/susgen/blob/HEAD/src/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"448dfa384a102a35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}