{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frugal-lms-trained-to-invoke-symbolic-solvers","title":"Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning","arxiv_id":"2312.05571","date":"2023-12-09","proceeding":null,"authors":["Subhabrata Dutta","Joykirat Singh","Ishan Pandey","Sunny Manchanda","Soumen Chakrabarti","Tanmoy Chakraborty"],"abstract":"Large Language Models (LLM) exhibit zero-shot mathematical reasoning capacity as a behavior emergent with scale, commonly manifesting as chain-of-thoughts (CoT) reasoning. However, multiple empirical findings suggest that this prowess is exclusive to LLMs with exorbitant sizes (beyond 50 billion parameters). Meanwhile, educational neuroscientists suggest that symbolic algebraic manipulation be introduced around the same time as arithmetic word problems to modularize language-to-formulation, symbolic manipulation of the formulation, and endgame arithmetic. In this paper, we start with the hypothesis that much smaller LMs, which are weak at multi-step reasoning, can achieve reasonable arithmetic reasoning if arithmetic word problems are posed as a formalize-then-solve task. In our architecture, which we call SYRELM, the LM serves the role of a translator to map natural language arithmetic questions into a formal language (FL) description. A symbolic solver then evaluates the FL expression to obtain the answer. A small frozen LM, equipped with an efficient low-rank adapter, is capable of generating FL expressions that incorporate natural language descriptions of the arithmetic problem (e.g., variable names and their purposes, formal expressions combining variables, etc.). We adopt policy-gradient reinforcement learning to train the adapted LM, informed by the non-differentiable symbolic solver. This marks a sharp departure from the recent development in tool-augmented LLMs, in which the external tools (e.g., calculator, Web search, etc.) are essentially detached from the learning phase of the LM. SYRELM shows massive improvements (e.g., +30.65 absolute point improvement in accuracy on the SVAMP dataset using GPT-J 6B model) over base LMs, while keeping our testbed easy to diagnose, interpret and within reach of most researchers.","url_abs":"https://arxiv.org/abs/2312.05571v2","url_pdf":"https://arxiv.org/pdf/2312.05571v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frugal-lms-trained-to-invoke-symbolic-solvers","repo_url":"https://github.com/joykirat18/syrelm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"math-word-problem-solving","task_name":"Math Word Problem Solving"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Vicuna (SYRELM)","rank_in_archive_order":144,"of":164,"metrics":{"Accuracy":"35.2","Parameters (Billion)":"13"},"uses_additional_data":true},{"leaderboard":"/sota/math-word-problem-solving-on-svamp","task":"Math Word Problem Solving","dataset":"SVAMP","model":"SYRELM (Vicuna 13B)","rank_in_archive_order":16,"of":26,"metrics":{"Execution Accuracy":"56.65"},"uses_additional_data":true},{"leaderboard":"/sota/math-word-problem-solving-on-svamp","task":"Math Word Problem Solving","dataset":"SVAMP","model":"SYRELM (GPT-J)","rank_in_archive_order":24,"of":26,"metrics":{"Execution Accuracy":"40.1"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.05571","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.05571"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joykirat18/syrelm","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"81acda882ca43bea","entry":"collater","repo":"joykirat18/syrelm","repo_kind":"official","path":"GPTJ/RL.py","file_url":"https://github.com/joykirat18/syrelm/blob/HEAD/GPTJ/RL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81acda882ca43bea"}},{"code_sha256_prefix":"3f8b9fe76f5a24b6","entry":"getGoldAnswer","repo":"joykirat18/syrelm","repo_kind":"official","path":"GPTJ/RL.py","file_url":"https://github.com/joykirat18/syrelm/blob/HEAD/GPTJ/RL.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f8b9fe76f5a24b6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}