{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/laiw-a-chinese-legal-large-language-models","title":"LAiW: A Chinese Legal Large Language Models Benchmark","arxiv_id":"2310.05620","date":"2023-10-09","proceeding":null,"authors":["Yongfu Dai","Duanyu Feng","Jimin Huang","Haochen Jia","Qianqian Xie","Yifang Zhang","Weiguang Han","Wei Tian","Hao Wang"],"abstract":"General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. However, the current evaluations of these LLMs in LegalAI are defined by the experts of computer science, lacking consistency with the logic of legal practice, making it difficult to judge their practical capabilities. To address this challenge, we are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice. To align with the thinking process of legal experts and legal practice (syllogism), we divide the legal capabilities of LLMs from easy to difficult into three levels: basic information retrieval, legal foundation inference, and complex legal application. Each level contains multiple tasks to ensure a comprehensive evaluation. Through automated evaluation of current general and legal domain LLMs on our benchmark, we indicate that these LLMs may not align with the logic of legal practice. LLMs seem to be able to directly acquire complex legal application capabilities but perform poorly in some basic tasks, which may pose obstacles to their practical application and acceptance by legal experts. To further confirm the complex legal application capabilities of current LLMs in legal application scenarios, we also incorporate human evaluation with legal experts. The results indicate that while LLMs may demonstrate strong performance, they still require reinforcement of legal logic.","url_abs":"https://arxiv.org/abs/2310.05620v2","url_pdf":"https://arxiv.org/pdf/2310.05620v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"laiw-a-chinese-legal-large-language-models","repo_url":"https://github.com/dai-shen/laiw","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.05620","atlas_url":"https://app.syntology.ai/?focus=2310.05620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dai-shen/laiw","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":8,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0928f497e20fb443","entry":"mean","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/metrics.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0928f497e20fb443"}},{"code_sha256_prefix":"57e732507bad40db","entry":"escaped_split","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/utils.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57e732507bad40db"}},{"code_sha256_prefix":"359c35fdcd7a0ee5","entry":"group","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/utils.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"359c35fdcd7a0ee5"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/base.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/base.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"0510e8cbb2c5a1e3","entry":"insert_space","repo":"dai-shen/laiw","repo_kind":"official","path":"precision/legal_metrics.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/precision/legal_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0510e8cbb2c5a1e3"}},{"code_sha256_prefix":"1213c8a6b0d1d84a","entry":"mcc","repo":"dai-shen/laiw","repo_kind":"official","path":"precision/legal_metrics.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/precision/legal_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1213c8a6b0d1d84a"}},{"code_sha256_prefix":"518b2201c66c2ad3","entry":"pop_stddev","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/metrics.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518b2201c66c2ad3"}},{"code_sha256_prefix":"efc4c3ac590c9737","entry":"process_text","repo":"dai-shen/laiw","repo_kind":"official","path":"src/tasks/utils.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/tasks/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efc4c3ac590c9737"}},{"code_sha256_prefix":"d40f0123e8a8f5c0","entry":"sample_stddev","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/metrics.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d40f0123e8a8f5c0"}},{"code_sha256_prefix":"cd999ad59caf84a7","entry":"simple_parse_args_string","repo":"dai-shen/laiw","repo_kind":"official","path":"src/financial-evaluation/lm_eval/utils.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/financial-evaluation/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd999ad59caf84a7"}},{"code_sha256_prefix":"df801139cecd0329","entry":"single_chat","repo":"dai-shen/laiw","repo_kind":"official","path":"src/chatlm.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/chatlm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df801139cecd0329"}},{"code_sha256_prefix":"5a6e0b41ccbd8a98","entry":"evaluate","repo":"dai-shen/laiw","repo_kind":"official","path":"src/interface.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/interface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a6e0b41ccbd8a98"}},{"code_sha256_prefix":"d8c48497dd9cc819","entry":"generate_prompt","repo":"dai-shen/laiw","repo_kind":"official","path":"src/interface.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/interface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8c48497dd9cc819"}},{"code_sha256_prefix":"9d95c14130837ee4","entry":"make_table","repo":"dai-shen/laiw","repo_kind":"official","path":"src/evaluator.py","file_url":"https://github.com/dai-shen/laiw/blob/HEAD/src/evaluator.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d95c14130837ee4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}