{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-the-compositional-deficiency-of","title":"Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning","arxiv_id":"2405.06680","date":"2024-05-05","proceeding":null,"authors":["Jun Zhao","Jingqi Tong","Yurong Mou","Ming Zhang","Qi Zhang","Xuanjing Huang"],"abstract":"Human cognition exhibits systematic compositionality, the algebraic ability to generate infinite novel combinations from finite learned components, which is the key to understanding and reasoning about complex logic. In this work, we investigate the compositionality of large language models (LLMs) in mathematical reasoning. Specifically, we construct a new dataset \\textsc{MathTrap} by introducing carefully designed logical traps into the problem descriptions of MATH and GSM8K. Since problems with logical flaws are quite rare in the real world, these represent \"unseen\" cases to LLMs. Solving these requires the models to systematically compose (1) the mathematical knowledge involved in the original problems with (2) knowledge related to the introduced traps. Our experiments show that while LLMs possess both components of requisite knowledge, they do not \\textbf{spontaneously} combine them to handle these novel cases. We explore several methods to mitigate this deficiency, such as natural language prompts, few-shot demonstrations, and fine-tuning. Additionally, we test the recently released OpenAI o1 model and find that human-like `slow thinking' helps improve the compositionality of LLMs. Overall, systematic compositionality remains an open challenge for large language models.","url_abs":"https://arxiv.org/abs/2405.06680v4","url_pdf":"https://arxiv.org/pdf/2405.06680v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-the-compositional-deficiency-of","repo_url":"https://github.com/tongjingqi/MathTrap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.06680","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.06680"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tongjingqi/MathTrap","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_draft_wrong":4,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0fe9cf6c00ef56ea","entry":"is_number","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/eval_GSM8K_category.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/eval_GSM8K_category.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0fe9cf6c00ef56ea"}},{"code_sha256_prefix":"66b3b5450182da3a","entry":"batch_data","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/eval_GSM8K_category.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/eval_GSM8K_category.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66b3b5450182da3a"}},{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"train_math.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/train_math.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"0b14c648516c38a7","entry":"last_boxed_only_string","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/util.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b14c648516c38a7"}},{"code_sha256_prefix":"51c01dd0d033f660","entry":"preprocess","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"train_math.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/train_math.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51c01dd0d033f660"}},{"code_sha256_prefix":"0bb621c7242d2fbe","entry":"read_json","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"GPT4_eval/get_response_example.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/GPT4_eval/get_response_example.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bb621c7242d2fbe"}},{"code_sha256_prefix":"f3bbe264b05aadd3","entry":"remove_boxed","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/eval_MATH_category2.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/eval_MATH_category2.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3bbe264b05aadd3"}},{"code_sha256_prefix":"8720975b15d076e8","entry":"last_boxed_only","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/util.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8720975b15d076e8"}},{"code_sha256_prefix":"abbd983f4f37a1d5","entry":"make_supervised_data_module","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"train_math.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/train_math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abbd983f4f37a1d5"}},{"code_sha256_prefix":"0bbf876ced018c83","entry":"only_until_first_boxed_from_tokens","repo":"tongjingqi/MathTrap","repo_kind":"official","path":"eval/util.py","file_url":"https://github.com/tongjingqi/MathTrap/blob/HEAD/eval/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bbf876ced018c83"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}