Formal Logic

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

3 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

20 shown of 20 papers with code (73 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic
  11 Aug 2023 3 repositories listed Syntology ran 0 of 2 samples · 2 unverified
  We rethink this and adopt a well-grounded set of deduction rules based on formal logic theory, which can derive any other deduction rules when combined in a multistep way.
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
  8 Dec 2021 3 repositories listed
  Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
• Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
  19 Nov 2024 2 repositories listed
  Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning.
• Learning Symbolic Rules for Reasoning in Quasi-Natural Language
  23 Nov 2021 2 repositories listed
  In this work, we ask how we can build a rule-based system that can reason with natural language input but without the manual construction of rules.
• Are LLMs Reliable Translators of Logical Reasoning Across Lexically Diversified Contexts?
  5 Jun 2025 1 repository listed
  Building directly on the deficiencies identified through our benchmark, we propose a new method, MenTaL, to address this limitation.
• Training Step-Level Reasoning Verifiers with Formal Verification Tools
  21 May 2025 1 repository listed
  Although this data synthesis is feasible only for tasks compatible with formal verification, we observe that LLM-based PRMs trained on our dataset exhibit cross-task generalization, improving verification across diverse…
• CoT-TL: Low-Resource Temporal Knowledge Representation of Planning Instructions Using Chain-of-Thought Reasoning
  21 Oct 2024 1 repository listed
  Finally, we validate CoT-TL's practicality by integrating it into a QuadCopter for multi-step drone planning based on natural language instructions.
• Autoformalization of Game Descriptions using Large Language Models
  18 Sep 2024 1 repository listed
  We evaluate the framework using GPT-4o and a dataset of natural language problem descriptions, achieving 98% syntactic correctness and 88% semantic correctness.
• Towards Logically Sound Natural Language Reasoning with Logic-Enhanced Language Model Agents
  28 Aug 2024 1 repository listed
  Large language models (LLMs) are increasingly explored as general-purpose reasoners, particularly in agentic contexts.
• Software Engineering Methods For AI-Driven Deductive Legal Reasoning
  15 Apr 2024 1 repository listed
  The recent proliferation of generative artificial intelligence (AI) technologies such as pre-trained large language models (LLMs) has opened up new frontiers in computational law.
• RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic
  3 Aug 2023 1 repository listed
  We present a comprehensive study of chart visual question-answering(QA) task, to address the challenges faced in comprehending and extracting data from chart visualizations within documents.
• Leveraging Large Language Models to Generate Answer Set Programs
  15 Jul 2023 1 repository listed Syntology ran 2 of 3 samples · 1 unverified
  Specifically, we employ an LLM to transform natural language descriptions of logic puzzles into answer set programs.
• Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation
  24 May 2023 1 repository listed Syntology ran 0 of 5 samples · 5 unverified
  Translating natural language sentences to first-order logic (NL-FOL translation) is a longstanding challenge in the NLP and formal logic literature.
• Nanopublication-Based Semantic Publishing and Reviewing: A Field Study with Formalization Papers
  3 Mar 2022 1 repository listed
  With the rapidly increasing amount of scientific literature, it is getting continuously more difficult for researchers in different disciplines to be updated with the recent findings in their field of study.
• Design of quantum optical experiments with logic artificial intelligence
  27 Sep 2021 1 repository listed
  In this work, we propose the use of logic AI for the design of optical quantum experiments.
• Satisfiability and Containment of Recursive SHACL
  30 Aug 2021 1 repository listed
  Using this logic we provide a detailed map of (un)decidability and complexity results for the satisfiability and containment decision problems for different SHACL fragments.
• Negation in Cognitive Reasoning
  23 Dec 2020 1 repository listed
  In this paper, we describe an effective procedure to determine the negated event or property in order to replace it by its inverse.
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
  21 Aug 2020 1 repository listed
  In this paper we show, first, that the entire theory of partial information decomposition can be derived from considerations of elementary parthood relationships between information contributions.
• Leveraging Rust types for modular specification and verification
  10 Oct 2019 1 repository listed
  In this paper, we present a novel verification technique that leverages Rust's type system to greatly simplify the specification and verification of system software written in Rust.
• Lectures on Jacques Herbrand as a Logician
  26 Feb 2009 1 repository listed
  We give some lectures on the work on formal logic of Jacques Herbrand, and sketch his life and his influence on automated theorem proving.

Syntology lines on 3 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.