{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/legalbench-rag-a-benchmark-for-retrieval","title":"LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain","arxiv_id":"2408.10343","date":"2024-08-19","proceeding":null,"authors":["Nicholas Pipitone","Ghita Houir Alami"],"abstract":"Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabilities of Large Language Models (LLMs) in the legal domain, but there is a critical gap in evaluating the retrieval component of RAG systems. To address this, we introduce LegalBench-RAG, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space. LegalBench-RAG emphasizes precise retrieval by focusing on extracting minimal, highly relevant text segments from legal documents. These highly relevant snippets are preferred over retrieving document IDs, or large sequences of imprecise chunks, both of which can exceed context window limitations. Long context windows cost more to process, induce higher latency, and lead LLMs to forget or hallucinate information. Additionally, precise results allow LLMs to generate citations for the end user. The LegalBench-RAG benchmark is constructed by retracing the context used in LegalBench queries back to their original locations within the legal corpus, resulting in a dataset of 6,858 query-answer pairs over a corpus of over 79M characters, entirely human-annotated by legal experts. We also introduce LegalBench-RAG-mini, a lightweight version for rapid iteration and experimentation. By providing a dedicated benchmark for legal retrieval, LegalBench-RAG serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain. The LegalBench-RAG dataset is publicly available at https://github.com/zeroentropy-cc/legalbenchrag.","url_abs":"https://arxiv.org/abs/2408.10343v1","url_pdf":"https://arxiv.org/pdf/2408.10343v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"legalbench-rag-a-benchmark-for-retrieval","repo_url":"https://github.com/zeroentropy-cc/legalbenchrag","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.10343","atlas_url":"https://app.syntology.ai/?focus=2408.10343","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.10343"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zeroentropy-cc/legalbenchrag","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"827171693ba8c487","entry":"avg","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/run_benchmark.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/run_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"827171693ba8c487"}},{"code_sha256_prefix":"0dee884215e4f10f","entry":"extract_quote_span","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/generate/generate_cuad.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/generate/generate_cuad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dee884215e4f10f"}},{"code_sha256_prefix":"157723f9b344254d","entry":"filename_pdf_to_text","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/generate/generate_cuad.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/generate/generate_cuad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"157723f9b344254d"}},{"code_sha256_prefix":"e47846e82d31718f","entry":"get_call_cache_key","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/utils/ai.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/utils/ai.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e47846e82d31718f"}},{"code_sha256_prefix":"3ecbffcfc211c000","entry":"get_contract_name_from_mae","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/generate/generate_maud.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/generate/generate_maud.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ecbffcfc211c000"}},{"code_sha256_prefix":"2165a33e822d6c61","entry":"serialize_f32","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/methods/baseline.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/methods/baseline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2165a33e822d6c61"}},{"code_sha256_prefix":"a1b887bfd0f75042","entry":"sort_and_merge_spans","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/benchmark_types.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/benchmark_types.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1b887bfd0f75042"}},{"code_sha256_prefix":"83f6d55fd211f2d3","entry":"get_contract_text","repo":"zeroentropy-cc/legalbenchrag","repo_kind":"official","path":"legalbenchrag/generate/generate_maud.py","file_url":"https://github.com/zeroentropy-cc/legalbenchrag/blob/HEAD/legalbenchrag/generate/generate_maud.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83f6d55fd211f2d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}