Legal Reasoning

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 30 papers with code (92 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• GPT-4 Technical Report
  15 Mar 2023 11 repositories listed Syntology ran 2 of 5 samples · 3 unverified · 1 pointer-only (licence)
  We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.
• DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services
  20 Sep 2023 2 repositories listed Syntology ran 5 of 6 samples · 1 unverified
  We propose DISC-LawLLM, an intelligent legal system utilizing large language models (LLMs) to provide a wide range of legal services.
• One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
  15 Jun 2023 2 repositories listed
  Our benchmark contains diverse datasets from the Swiss legal system, allowing for a comprehensive study of the underlying non-English, inherently multilingual legal system.
• Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws
  2 Jun 2025 1 repository listed
  This study uses Jordanian law as a case study to explore the fine-tuning of the Llama-3.
• LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements
  29 May 2025 1 repository listed
  Health, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions.
• Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use
  4 May 2025 1 repository listed
  This paper presents a domain-specific implementation of Retrieval-Augmented Generation (RAG) tailored to the Fair Use Doctrine in U.
• CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation
  25 Feb 2025 1 repository listed
  To the best of our knowledge, CaseGen is the first benchmark designed to evaluate LLMs in the context of legal case document generation.
• JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning
  24 Feb 2025 1 repository listed Syntology ran 0 of 15 samples · 15 unverified
  The Four-Element Theory is a fundamental framework in criminal law, defining the constitution of crime through four dimensions: Subject, Object, Subjective aspect, and Objective aspect.
• NitiBench: A Comprehensive Studies of LLM Frameworks Capabilities for Thai Legal Question Answering
  15 Feb 2025 1 repository listed
  The application of large language models (LLMs) in the legal domain holds significant potential for information retrieval and question answering, yet Thai legal QA systems face challenges due to a lack of standardized…
• Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning
  11 Feb 2025 1 repository listed
  Large Language Models (LLMs) have achieved impressive results across numerous domains, yet they experience notable deficiencies in legal question-answering tasks.
• LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM
  10 Feb 2025 1 repository listed
  To address these limitations, we study data generation for legal reasoning to improve the legal reasoning performance of open-source LLMs with the help of proprietary LLMs.
• Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning
  8 Feb 2025 1 repository listed
  We then develop an LLM-based automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs.
• Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning
  16 Oct 2024 1 repository listed
  We remark that existing works investigate the phenomenon of weak-to-strong generation in analogous setup (i.
• Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models
  11 Oct 2024 1 repository listed
  Here, we introduce KBL, a benchmark for assessing the Korean legal language understanding of LLMs, consisting of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the Korean…
• Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaboration
  3 Oct 2024 1 repository listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  Large Language Models (LLMs) could struggle to fully understand legal theories and perform complex legal reasoning tasks.
• LeKUBE: A Legal Knowledge Update BEnchmark
  19 Jul 2024 1 repository listed
  Existing benchmarks for evaluating knowledge update methods are mostly designed for the open domain and cannot address the specific challenges of the legal domain, such as the nuanced application of new legal knowledge,…
• Software Engineering Methods For AI-Driven Deductive Legal Reasoning
  15 Apr 2024 1 repository listed
  The recent proliferation of generative artificial intelligence (AI) technologies such as pre-trained large language models (LLMs) has opened up new frontiers in computational law.
• LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain
  2 Apr 2024 1 repository listed
  We publish LawInstruct as a resource for further study of instruction tuning in the legal domain.
• ECtHR-PCR: A Dataset for Precedent Understanding and Prior Case Retrieval in the European Court of Human Rights
  31 Mar 2024 1 repository listed
  In common law jurisdictions, legal practitioners rely on precedents to construct arguments, in line with the doctrine of \emph{stare decisis}.
• Chain of Logic: Rule-Based Reasoning with Large Language Models
  16 Feb 2024 1 repository listed
  We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to…
• TMID: A Comprehensive Real-world Dataset for Trademark Infringement Detection in E-Commerce
  8 Dec 2023 1 repository listed
  Annually, e-commerce platforms incur substantial financial losses due to trademark infringements, making it crucial to identify and mitigate potential legal risks tied to merchant information registered to the platforms.
• Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
  27 Oct 2023 1 repository listed Syntology ran 4 of 4 samples · 0 unverified · 4 pointer-only (licence)
  Our findings generally sound a note of caution in the use of generative LMs on complex tasks without fine-tuning and point to the continued relevance of human annotation-intensive classification methods.
• Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?
  23 Oct 2023 1 repository listed
  Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations.
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction
  18 Oct 2023 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain.
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  20 Aug 2023 1 repository listed Syntology ran 1 of 3 samples · 2 unverified · 3 pointer-only (licence)
  The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform?
• LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
  13 Sep 2022 1 repository listed
  Finally-inspired by the Open Science movement-we make a call for the legal and computer science communities to join our efforts by contributing new tasks.
• Claim Extraction and Law Matching for COVID-19-related Legislation
  1 Jun 2022 1 repository listed
  We investigate an automated approach to extract legal claims from news articles and to match the claims with their corresponding applicable laws.
• Designing Normative Theories for Ethical and Legal Reasoning: LogiKEy Framework, Methodology, and Tool Support
  25 Mar 2019 1 repository listed
  A framework and methodology---termed LogiKEy---for the design and engineering of ethical reasoners, normative theories and deontic logics is presented.
• Passing the Brazilian OAB Exam: data preparation and some experiments
  14 Dec 2017 1 repository listed
  In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams.
• Causality and Responsibility for Formal Verification and Beyond
  29 Aug 2016 1 repository listed
  The theory of actual causality, defined by Halpern and Pearl, and its quantitative measure - the degree of responsibility - was shown to be extremely useful in various areas of computer science due to a good match…

Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.