{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-llms-solve-longer-math-word-problems","title":"Can LLMs Solve longer Math Word Problems Better?","arxiv_id":"2405.14804","date":"2024-05-23","proceeding":null,"authors":["Xin Xu","Tong Xiao","Zitong Chao","Zhenya Huang","Can Yang","Yang Wang"],"abstract":"Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-School Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of LLMs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with open-source LLMs reveals a general deficiency in CoLeG. To alleviate these issues, we propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce a new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop a novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy, showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current LLMs and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.","url_abs":"https://arxiv.org/abs/2405.14804v2","url_pdf":"https://arxiv.org/pdf/2405.14804v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-llms-solve-longer-math-word-problems","repo_url":"https://github.com/xinxu-ustc/coleg-math","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.14804","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14804"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LYH-YF/MWPToolkit","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-research-datasets/GSM-IC","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xinxu-ustc/coleg-math","reach":null}],"summary":{"ran":5,"ran_draft_wrong":6,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"found_in_text":{"samples":6,"ran":5,"repositories":1},"community":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"70fde71e0eca9262","entry":"DataArguments","repo":"xinxu-ustc/coleg-math","repo_kind":"official","path":"proprietary-llms/prompt_generator.py","file_url":"https://github.com/xinxu-ustc/coleg-math/blob/HEAD/proprietary-llms/prompt_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70fde71e0eca9262"}},{"code_sha256_prefix":"2c3131374c3b000f","entry":"MaskSymbol","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c3131374c3b000f"}},{"code_sha256_prefix":"2c71295aed1296d7","entry":"NumMask","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c71295aed1296d7"}},{"code_sha256_prefix":"48332bf292c7a698","entry":"PromptGenerator","repo":"xinxu-ustc/coleg-math","repo_kind":"official","path":"proprietary-llms/prompt_generator.py","file_url":"https://github.com/xinxu-ustc/coleg-math/blob/HEAD/proprietary-llms/prompt_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48332bf292c7a698"}},{"code_sha256_prefix":"ac123f87180f8ea4","entry":"RunningArguments","repo":"xinxu-ustc/coleg-math","repo_kind":"official","path":"proprietary-llms/prompt_generator.py","file_url":"https://github.com/xinxu-ustc/coleg-math/blob/HEAD/proprietary-llms/prompt_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac123f87180f8ea4"}},{"code_sha256_prefix":"66b3b5450182da3a","entry":"batch_data","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"66b3b5450182da3a"}},{"code_sha256_prefix":"ecae39d7fc9bf490","entry":"eval_with_timeout","repo":"openai/grade-school-math","repo_kind":"community","path":"grade_school_math/calculator.py","file_url":"https://github.com/openai/grade-school-math/blob/HEAD/grade_school_math/calculator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecae39d7fc9bf490"}},{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"c9f3d4816009a6e4","entry":"lists2dict","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9f3d4816009a6e4"}},{"code_sha256_prefix":"f3bbe264b05aadd3","entry":"remove_boxed","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f3bbe264b05aadd3"}},{"code_sha256_prefix":"14cc18e6c2bedaa7","entry":"seg_and_tag_math23k","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14cc18e6c2bedaa7"}},{"code_sha256_prefix":"7e32bb1abfcffea8","entry":"str2float","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e32bb1abfcffea8"}},{"code_sha256_prefix":"3080becc78cd2db7","entry":"number_transfer_math23k","repo":"LYH-YF/MWPToolkit","repo_kind":"found_in_text","path":"mwptoolkit/utils/preprocess_tools.py","file_url":"https://github.com/LYH-YF/MWPToolkit/blob/HEAD/mwptoolkit/utils/preprocess_tools.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3080becc78cd2db7"}},{"code_sha256_prefix":"f982e7500a211eee","entry":"sample","repo":"openai/grade-school-math","repo_kind":"community","path":"grade_school_math/calculator.py","file_url":"https://github.com/openai/grade-school-math/blob/HEAD/grade_school_math/calculator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f982e7500a211eee"}},{"code_sha256_prefix":"7eaccadc036430c1","entry":"use_calculator","repo":"openai/grade-school-math","repo_kind":"community","path":"grade_school_math/calculator.py","file_url":"https://github.com/openai/grade-school-math/blob/HEAD/grade_school_math/calculator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7eaccadc036430c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}