{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-10186","title":"LegalCiteBench: Evaluating Citation Reliability in Legal Language Models","arxiv_id":"2605.10186","date":"2026-05-11","proceeding":null,"authors":["Sijia Chen","Hang Yin","Shunfan Zhou"],"abstract":"Large language models (LLMs) are increasingly integrated into legal drafting and research workflows, where incorrect citations or fabricated precedents can cause serious professional harm. Existing legal benchmarks largely emphasize statutory reasoning, contract understanding, or general legal question answering, but they do not directly study a central common-law failure mode: when asked to provide case authorities without external grounding, models may return plausible-looking but incorrect citations or cases. We introduce LegalCiteBench, a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models. LegalCiteBench contains approximately 24K evaluation instances constructed from 1,000 real U.S. judicial opinions from the Case Law Access Project. The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction. Across 21 LLMs, exact citation recovery remains highly challenging in this closed-book setting: even the strongest models score below 7/100 on citation retrieval and completion. Within the evaluated models, scale and legal-domain pretraining provide limited gains and do not resolve this difficulty. Models also frequently provide concrete but incorrect or low-overlap authorities under our evaluation protocol, with Misleading Answer Rates (MAR) exceeding 94% for 20 of 21 evaluated models on retrieval-heavy tasks. A prompt-only abstention experiment shows that explicit uncertainty instructions reduce some confident fabrication but do not improve citation correctness. LegalCiteBench is intended as a diagnostic framework for studying authority generation failures, verification behavior, and abstention when external grounding is absent, incomplete, or bypassed.","url_abs":"https://arxiv.org/abs/2605.10186","url_pdf":"https://arxiv.org/pdf/2605.10186","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.10186","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.10186"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Sijia711/LegalCiteBench","reach":null}],"summary":{"ran":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5d93b1456f216ca5","entry":"format_ground_truth","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d93b1456f216ca5"}},{"code_sha256_prefix":"a99a765f3eb79781","entry":"get_system_prompt","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/eval/eval_model.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/eval/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a99a765f3eb79781"}},{"code_sha256_prefix":"c06818718c5e5355","entry":"load_jsonl","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c06818718c5e5355"}},{"code_sha256_prefix":"c914ed2a7e46aeff","entry":"load_models_from_config","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/eval/llm_judge.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/eval/llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c914ed2a7e46aeff"}},{"code_sha256_prefix":"6dc1bbd43387da99","entry":"make_output_key","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/eval/eval_model.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/eval/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6dc1bbd43387da99"}},{"code_sha256_prefix":"6b355005d540fac5","entry":"sanitize_model_name","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/eval/eval_model.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/eval/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b355005d540fac5"}},{"code_sha256_prefix":"4d1b68ae51e608b7","entry":"sanitize_model_name","repo":"Sijia711/LegalCiteBench","repo_kind":"found_in_text","path":"legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","file_url":"https://github.com/Sijia711/LegalCiteBench/blob/HEAD/legal-citation-benchmark-clean/analysis/run_prompt_mitigation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d1b68ae51e608b7"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}