{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/symbolic-working-memory-enhances-language","title":"Symbolic Working Memory Enhances Language Models for Complex Rule Application","arxiv_id":"2408.13654","date":"2024-08-24","proceeding":null,"authors":["Siyuan Wang","Zhongyu Wei","Yejin Choi","Xiang Ren"],"abstract":"Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially. Our preliminary analysis shows that while LLMs excel in single-step rule application, their performance drops significantly in multi-step scenarios due to the challenge in rule grounding. It requires anchoring the applicable rule and supporting facts at each step, amidst multiple input rules, facts, and inferred facts. To address this, we propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. The memory stores facts and rules in both natural language and symbolic forms, enabling precise tracking. Utilizing this memory, our framework iteratively performs symbolic rule grounding and LLM-based rule implementation. The former matches predicates and variables of symbolic rules and facts to ground applicable rules at each step. Experiments indicate our framework's effectiveness in rule application and its robustness across various steps and settings~\\footnote{Code and data are available at \\url{https://github.com/SiyuanWangw/RuleApplication}.}.","url_abs":"https://arxiv.org/abs/2408.13654v1","url_pdf":"https://arxiv.org/pdf/2408.13654v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"symbolic-working-memory-enhances-language","repo_url":"https://github.com/siyuanwangw/ruleapplication","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.13654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.13654"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/siyuanwangw/ruleapplication","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"4e1c4ff342f96bc9","entry":"dispatch_openai_chat_requests","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/models/openai_models.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/models/openai_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e1c4ff342f96bc9"}},{"code_sha256_prefix":"94928eafb9afc8c7","entry":"dispatch_openai_prompt_requests","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/models/openai_models.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/models/openai_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94928eafb9afc8c7"}},{"code_sha256_prefix":"76bc3e42fa9a54a8","entry":"find_individual","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/evaluation_utils.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76bc3e42fa9a54a8"}},{"code_sha256_prefix":"83acc70202392615","entry":"load_prompt_fact_init","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/utils.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83acc70202392615"}},{"code_sha256_prefix":"e19b01a83156528a","entry":"load_prompt_final_predict","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/final_predict.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/final_predict.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e19b01a83156528a"}},{"code_sha256_prefix":"6f2aab0105d6de70","entry":"load_prompt_rule_implementation","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/utils.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f2aab0105d6de70"}},{"code_sha256_prefix":"d8e076b666a81e08","entry":"load_raw_dataset","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/evaluate.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8e076b666a81e08"}},{"code_sha256_prefix":"1675e6f35d4f8ba4","entry":"parse_answer","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/evaluation_utils.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1675e6f35d4f8ba4"}},{"code_sha256_prefix":"7c03c8915ba5c271","entry":"post_process_fact","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/response_postprocess.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/response_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c03c8915ba5c271"}},{"code_sha256_prefix":"11e3e3ef0f5ff617","entry":"post_process_implement","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/response_postprocess.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/response_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11e3e3ef0f5ff617"}},{"code_sha256_prefix":"29bba7fd8c210dad","entry":"post_process_rule","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/response_postprocess.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/response_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29bba7fd8c210dad"}},{"code_sha256_prefix":"bf4cfef1d50731f1","entry":"evaluate_performance","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/evaluate.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf4cfef1d50731f1"}},{"code_sha256_prefix":"f6f5e08416631644","entry":"load_prompt_rule_init","repo":"siyuanwangw/ruleapplication","repo_kind":"official","path":"Src/utils.py","file_url":"https://github.com/siyuanwangw/ruleapplication/blob/HEAD/Src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6f5e08416631644"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}