{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-comprehension-language-models-with","title":"Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks","arxiv_id":"2304.01665","date":"2023-04-04","proceeding":null,"authors":["Yixuan Weng","Minjun Zhu","Fei Xia","Bin Li","Shizhu He","Kang Liu","Jun Zhao"],"abstract":"Language models' (LMs) proficiency in handling deterministic symbolic reasoning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To endow LMs with genuine rule comprehension abilities, we propose \"Neural Comprehension\" - a framework that synergistically integrates compiled neural networks (CoNNs) into the standard transformer architecture. CoNNs are neural modules designed to explicitly encode rules through artificially generated attention weights. By incorporating CoNN modules, the Neural Comprehension framework enables LMs to accurately and robustly execute rule-intensive symbolic tasks. Extensive experiments demonstrate the superiority of our approach over existing techniques in terms of length generalization, efficiency, and interpretability for symbolic operations. Furthermore, it can be applied to LMs across different model scales, outperforming tool-calling methods in arithmetic reasoning tasks while maintaining superior inference efficiency. Our work highlights the potential of seamlessly unifying explicit rule learning via CoNNs and implicit pattern learning in LMs, paving the way for true symbolic comprehension capabilities.","url_abs":"https://arxiv.org/abs/2304.01665v3","url_pdf":"https://arxiv.org/pdf/2304.01665v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-comprehension-language-models-with","repo_url":"https://github.com/wengsyx/neural-comprehension","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-comprehension-language-models-with","repo_url":"https://github.com/wengsyx/controllm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-comprehension-language-models-with","repo_url":"https://github.com/wengsyx/lmtuner","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.01665","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.01665"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wengsyx/controllm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wengsyx/neural-comprehension","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wengsyx/lmtuner","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":2,"repositories":1},"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"2fdba0a9c7efc372","entry":"AttnWrapper","repo":"wengsyx/controllm","repo_kind":"listed","path":"ControlLM/falcon/model.py","file_url":"https://github.com/wengsyx/controllm/blob/HEAD/ControlLM/falcon/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fdba0a9c7efc372"}},{"code_sha256_prefix":"fbe4b9e916cb63e1","entry":"BlockOutputWrapper","repo":"wengsyx/controllm","repo_kind":"listed","path":"ControlLM/falcon/model.py","file_url":"https://github.com/wengsyx/controllm/blob/HEAD/ControlLM/falcon/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fbe4b9e916cb63e1"}},{"code_sha256_prefix":"dd791f54ccd3df38","entry":"CoNNEmbeddings","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd791f54ccd3df38"}},{"code_sha256_prefix":"9df2036489261543","entry":"TransformerEncoder","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9df2036489261543"}},{"code_sha256_prefix":"756d705bfd684fbd","entry":"add_vector_after_position","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"756d705bfd684fbd"}},{"code_sha256_prefix":"0d3b9b4818b3579c","entry":"CoNNConfig","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d3b9b4818b3579c"}},{"code_sha256_prefix":"67acc76bace2a9f7","entry":"CoNNModel","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67acc76bace2a9f7"}},{"code_sha256_prefix":"5944800bcff3213c","entry":"CoNNPreTrainedModel","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5944800bcff3213c"}},{"code_sha256_prefix":"d533386a26ebc88e","entry":"FalconControlLM","repo":"wengsyx/controllm","repo_kind":"listed","path":"ControlLM/falcon/model.py","file_url":"https://github.com/wengsyx/controllm/blob/HEAD/ControlLM/falcon/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d533386a26ebc88e"}},{"code_sha256_prefix":"3e1a3d7b3bf9ed48","entry":"MultiheadAttention","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e1a3d7b3bf9ed48"}},{"code_sha256_prefix":"17a235d61eb3ac7a","entry":"TransformerEncoderLayer","repo":"wengsyx/neural-comprehension","repo_kind":"official","path":"NeuralCom/CoNN/modeling_conn.py","file_url":"https://github.com/wengsyx/neural-comprehension/blob/HEAD/NeuralCom/CoNN/modeling_conn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17a235d61eb3ac7a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}