Browse State-of-the-Art › Legal Reasoning
Legal Reasoning
30 papers with code · 2 benchmarks · 0 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| LegalBench (Issue-spotting) (3 rows) | GPT-4 | — | — | — | Compare |
| LegalBench (Rule-recall) (1 row) | GPT-4 | GPT-4 Technical Report | code | Syntology community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 92 · with a code link: 30 · where Syntology ran a sample: 7 (6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (7 of 92 tagged: 6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 92 of 92, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
15 Mar 2023 11 repositories listed Syntology community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 1 pointer-only (licence)We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.
-
20 Sep 2023 2 repositories listed Syntology official (archive's flag): 3 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)We propose DISC-LawLLM, an intelligent legal system utilizing large language models (LLMs) to provide a wide range of legal services.
-
15 Jun 2023 2 repositories listedOur benchmark contains diverse datasets from the Swiss legal system, allowing for a comprehensive study of the underlying non-English, inherently multilingual legal system.
-
2 Jun 2025 1 repository listedThis study uses Jordanian law as a case study to explore the fine-tuning of the Llama-3.
-
29 May 2025 1 repository listedHealth, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions.
-
4 May 2025 1 repository listedThis paper presents a domain-specific implementation of Retrieval-Augmented Generation (RAG) tailored to the Fair Use Doctrine in U.
-
25 Feb 2025 1 repository listedTo the best of our knowledge, CaseGen is the first benchmark designed to evaluate LLMs in the context of legal case document generation.
-
24 Feb 2025 1 repository listed Syntology official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 15 harvested samples)The Four-Element Theory is a fundamental framework in criminal law, defining the constitution of crime through four dimensions: Subject, Object, Subjective aspect, and Objective aspect.
-
15 Feb 2025 1 repository listedThe application of large language models (LLMs) in the legal domain holds significant potential for information retrieval and question answering, yet Thai legal QA systems face challenges due to a lack of standardized…
-
11 Feb 2025 1 repository listedLarge Language Models (LLMs) have achieved impressive results across numerous domains, yet they experience notable deficiencies in legal question-answering tasks.
-
10 Feb 2025 1 repository listedTo address these limitations, we study data generation for legal reasoning to improve the legal reasoning performance of open-source LLMs with the help of proprietary LLMs.
-
8 Feb 2025 1 repository listedWe then develop an LLM-based automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs.
-
16 Oct 2024 1 repository listedWe remark that existing works investigate the phenomenon of weak-to-strong generation in analogous setup (i. e., binary classification), rather than practical alignment-relevant tasks (e. g., safety).
-
11 Oct 2024 1 repository listedHere, we introduce KBL, a benchmark for assessing the Korean legal language understanding of LLMs, consisting of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the Korean…
-
3 Oct 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)Large Language Models (LLMs) could struggle to fully understand legal theories and perform complex legal reasoning tasks.
-
19 Jul 2024 1 repository listedExisting benchmarks for evaluating knowledge update methods are mostly designed for the open domain and cannot address the specific challenges of the legal domain, such as the nuanced application of new legal knowledge…
-
15 Apr 2024 1 repository listedThe recent proliferation of generative artificial intelligence (AI) technologies such as pre-trained large language models (LLMs) has opened up new frontiers in computational law.
-
2 Apr 2024 1 repository listedWe publish LawInstruct as a resource for further study of instruction tuning in the legal domain.
-
31 Mar 2024 1 repository listedIn common law jurisdictions, legal practitioners rely on precedents to construct arguments, in line with the doctrine of stare decisis.
-
16 Feb 2024 1 repository listedWe introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to…
-
8 Dec 2023 1 repository listedAnnually, e-commerce platforms incur substantial financial losses due to trademark infringements, making it crucial to identify and mitigate potential legal risks tied to merchant information registered to the platforms.
-
27 Oct 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)Our findings generally sound a note of caution in the use of generative LMs on complex tasks without fine-tuning and point to the continued relevance of human annotation-intensive classification methods.
-
23 Oct 2023 1 repository listedEach scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations.
-
18 Oct 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain.
-
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models20 Aug 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples) · 3 pointer-only (licence)The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform?
-
13 Sep 2022 1 repository listedFinally-inspired by the Open Science movement-we make a call for the legal and computer science communities to join our efforts by contributing new tasks.
-
1 Jun 2022 1 repository listedWe investigate an automated approach to extract legal claims from news articles and to match the claims with their corresponding applicable laws.
-
25 Mar 2019 1 repository listedA framework and methodology---termed LogiKEy---for the design and engineering of ethical reasoners, normative theories and deontic logics is presented.
-
14 Dec 2017 1 repository listedIn Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams.
-
29 Aug 2016 1 repository listedThe theory of actual causality, defined by Halpern and Pearl, and its quantitative measure - the degree of responsibility - was shown to be extremely useful in various areas of computer science due to a good match…
-
IndianBailJudgments-1200: A Multi-Attribute Dataset for Legal NLP on Indian Bail Orders3 Jul 2025 0 repositories listed
-
Large Language Models Acing Chartered Accountancy26 Jun 2025 0 repositories listed
-
CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models5 Jun 2025 0 repositories listed
-
When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning4 Jun 2025 0 repositories listed
-
LEXam: Benchmarking Legal Reasoning on 340 Law Exams19 May 2025 0 repositories listed
-
SynLexLM: Scaling Legal LLMs with Synthetic Data and Curriculum Learning26 Apr 2025 0 repositories listed
-
Engineering the Law-Machine Learning Translation Problem: Developing Legally Aligned Models23 Apr 2025 0 repositories listed
-
Continual Pre-Training is (not) What You Need in Domain Adaption18 Apr 2025 0 repositories listed
-
KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding17 Apr 2025 0 repositories listed
-
An Explicit Syllogistic Legal Reasoning Framework for Large Language Models5 Apr 2025 0 repositories listed
-
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond20 Mar 2025 0 repositories listed
-
Adaptively profiling models with task elicitation3 Mar 2025 0 repositories listed
-
Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law24 Feb 2025 0 repositories listed
-
Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York13 Feb 2025 0 repositories listed
-
Artificial Intelligence and Legal Analysis: Implications for Legal Education and the Profession4 Feb 2025 0 repositories listed
-
Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains24 Jan 2025 0 repositories listed
-
Legal Evalutions and Challenges of Large Language Models15 Nov 2024 0 repositories listed
-
LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights17 Oct 2024 0 repositories listed
-
KRAG Framework for Enhancing LLMs in the Legal Domain10 Oct 2024 0 repositories listed
-
Using LLMs to Discover Legal Factors10 Oct 2024 0 repositories listed
-
LegiLM: A Fine-Tuned Legal Language Model for Data Compliance9 Sep 2024 0 repositories listed
-
LAPIS: Language Model-Augmented Police Investigation System19 Jul 2024 0 repositories listed
-
Formalising Anti-Discrimination Law in Automated Decision Systems29 Jun 2024 0 repositories listed
-
Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset with IRAC methodology19 Jun 2024 0 repositories listed
-
Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?16 Jun 2024 0 repositories listed
-
Explainable machine learning multi-label classification of Spanish legal judgements27 May 2024 0 repositories listed
-
PARAMANU-AYN: Pretrain from scratch or Continual Pretraining of LLMs for Legal Domain Adaptation?20 Mar 2024 0 repositories listed
-
Advancing Legal Reasoning: The Integration of AI to Navigate Complexities and Biases in Global Jurisprudence with Semi-Automated Arbitration Processes (SAAPs)6 Feb 2024 0 repositories listed
-
Aalap: AI Assistant for Legal & Paralegal Functions in India30 Jan 2024 0 repositories listed
-
Automated legal reasoning with discretion to act using s(LAW)25 Jan 2024 0 repositories listed
-
Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applications22 Nov 2023 0 repositories listed
-
Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?11 Aug 2023 0 repositories listed
-
Exploring the psychology of LLMs' Moral and Legal Reasoning2 Aug 2023 0 repositories listed
-
Argumentation Schemes for Blockchain Deanonymization26 May 2023 0 repositories listed
-
Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree5 May 2023 0 repositories listed
-
Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases1 Feb 2023 0 repositories listed
-
Law to Binary Tree -- An Formal Interpretation of Legal Natural Language16 Dec 2022 0 repositories listed
-
Legal Prompting: Teaching a Language Model to Think Like a Lawyer2 Dec 2022 0 repositories listed
-
Using Argumentation Schemes to Model Legal Reasoning1 Oct 2022 0 repositories listed
-
Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans14 Sep 2022 0 repositories listed
-
An Argumentation-Based Legal Reasoning Approach for DL-Ontology7 Sep 2022 0 repositories listed
-
Non-Determinism and the Lawlessness of Machine Learning Code23 Jun 2022 0 repositories listed
-
Searching for Legal Documents at Paragraph Level: Automating Label Generation and Use of an Extended Attention Mask for Boosting Neural Models of Semantic Similarity1 Nov 2021 0 repositories listed
-
Exploiting Domain-Specific Knowledge for Judgment Prediction Is No Panacea1 Sep 2021 0 repositories listed
-
New Algebraic Normative Theories for Ethical and Legal Reasoning in the LogiKEy Framework25 Jul 2021 0 repositories listed
-
AutoLAW: Augmented Legal Reasoning through Legal Precedent Prediction30 Jun 2021 0 repositories listed
-
Antitrust and Artificial Intelligence (AAI): Antitrust Vigilance Lifecycle and AI Legal Reasoning Autonomy23 Dec 2020 0 repositories listed
-
Legal Sentiment Analysis and Opinion Mining (LSAOM): Assimilating Advances in Autonomous AI Legal Reasoning2 Oct 2020 0 repositories listed
-
Legal Judgment Prediction (LJP) Amid the Advent of Autonomous AI Legal Reasoning29 Sep 2020 0 repositories listed
-
The Next Era of American Law Amid the Advent of Autonomous AI Legal Reasoning21 Sep 2020 0 repositories listed
-
Deconstructing Legal Text_Object Oriented Design in Legal Adjudication13 Sep 2020 0 repositories listed
-
AI and Legal Argumentation: Aligning the Autonomous Levels of AI Legal Reasoning11 Sep 2020 0 repositories listed
-
Robustness and Overcoming Brittleness of AI-Enabled Legal Micro-Directives: The Role of Autonomous Levels of AI Legal Reasoning31 Aug 2020 0 repositories listed
-
An Impact Model of AI on the Principles of Justice: Encompassing the Autonomous Levels of AI Legal Reasoning26 Aug 2020 0 repositories listed
-
Multidimensionality of Legal Singularity: Parametric Analysis and the Autonomous Levels of AI Legal Reasoning24 Aug 2020 0 repositories listed
-
Authorized and Unauthorized Practices of Law: The Role of Autonomous Levels of AI Legal Reasoning19 Aug 2020 0 repositories listed
-
Turing Test and the Practice of Law: The Role of Autonomous Levels of AI Legal Reasoning18 Aug 2020 0 repositories listed
-
An Ontological AI-and-Law Framework for the Autonomous Levels of AI Legal Reasoning4 Aug 2020 0 repositories listed
-
Modelling Value-oriented Legal Reasoning in LogiKEy23 Jun 2020 0 repositories listed
-
Proceedings First Workshop on Causal Reasoning for Embedded and safety-critical Systems Technologies26 Aug 2016 0 repositories listed
-
Automated Reasoning for Robot Ethics20 Feb 2015 0 repositories listed
-
Automated Reasoning in Deontic Logic18 Nov 2014 0 repositories listed
Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections