{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/linc-a-neurosymbolic-approach-for-logical","title":"LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers","arxiv_id":"2310.15164","date":"2023-10-23","proceeding":null,"authors":["Theo X. Olausson","Alex Gu","Benjamin Lipkin","Cedegao E. Zhang","Armando Solar-Lezama","Joshua B. Tenenbaum","Roger Levy"],"abstract":"Logical reasoning, i.e., deductively inferring the truth value of a conclusion from a set of premises, is an important task for artificial intelligence with wide potential impacts on science, mathematics, and society. While many prompting-based strategies have been proposed to enable Large Language Models (LLMs) to do such reasoning more effectively, they still appear unsatisfactory, often failing in subtle and unpredictable ways. In this work, we investigate the validity of instead reformulating such tasks as modular neurosymbolic programming, which we call LINC: Logical Inference via Neurosymbolic Computation. In LINC, the LLM acts as a semantic parser, translating premises and conclusions from natural language to expressions in first-order logic. These expressions are then offloaded to an external theorem prover, which symbolically performs deductive inference. Leveraging this approach, we observe significant performance gains on FOLIO and a balanced subset of ProofWriter for three different models in nearly all experimental conditions we evaluate. On ProofWriter, augmenting the comparatively small open-source StarCoder+ (15.5B parameters) with LINC even outperforms GPT-3.5 and GPT-4 with Chain-of-Thought (CoT) prompting by an absolute 38% and 10%, respectively. When used with GPT-4, LINC scores 26% higher than CoT on ProofWriter while performing comparatively on FOLIO. Further analysis reveals that although both methods on average succeed roughly equally often on this dataset, they exhibit distinct and complementary failure modes. We thus provide promising evidence for how logical reasoning over natural language can be tackled through jointly leveraging LLMs alongside symbolic provers. All corresponding code is publicly available at https://github.com/benlipkin/linc","url_abs":"https://arxiv.org/abs/2310.15164v2","url_pdf":"https://arxiv.org/pdf/2310.15164v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"linc-a-neurosymbolic-approach-for-logical","repo_url":"https://github.com/benlipkin/linc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.15164","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.15164"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/benlipkin/linc","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Yale-LILY/FOLIO","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"8ab896521f87794c","entry":"get_all_variables","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ab896521f87794c"}},{"code_sha256_prefix":"560845bb152e2176","entry":"FOLIOBase","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"560845bb152e2176"}},{"code_sha256_prefix":"b40071b137b0c945","entry":"OWAFOLTask","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b40071b137b0c945"}},{"code_sha256_prefix":"ed34cd2d0c6826c0","entry":"Task","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed34cd2d0c6826c0"}},{"code_sha256_prefix":"81c2ff6cb123d80a","entry":"convert_to_nltk_rep","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81c2ff6cb123d80a"}},{"code_sha256_prefix":"c6af10b5a9445eee","entry":"evaluate","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6af10b5a9445eee"}},{"code_sha256_prefix":"11f03965b940b736","entry":"reformat_fol","repo":"benlipkin/linc","repo_kind":"official","path":"eval/tasks/folio.py","file_url":"https://github.com/benlipkin/linc/blob/HEAD/eval/tasks/folio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11f03965b940b736"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}