{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llms-instead-of-human-judges-a-large-scale","title":"LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks","arxiv_id":"2406.18403","date":"2024-06-26","proceeding":null,"authors":["Anna Bavaresco","Raffaella Bernardi","Leonardo Bertolazzi","Desmond Elliott","Raquel Fernández","Albert Gatt","Esam Ghaleb","Mario Giulianelli","Michael Hanna","Alexander Koller","André F. T. Martins","Philipp Mondorf","Vera Neplenbroek","Sandro Pezzelle","Barbara Plank","David Schlangen","Alessandro Suglia","Aditya K Surikuchi","Ece Takmaz","Alberto Testoni"],"abstract":"There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators.","url_abs":"https://arxiv.org/abs/2406.18403v2","url_pdf":"https://arxiv.org/pdf/2406.18403v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llms-instead-of-human-judges-a-large-scale","repo_url":"https://github.com/dmg-illc/JUDGE-BENCH","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.18403","atlas_url":"https://app.syntology.ai/?focus=2406.18403","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.18403"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmg-illc/JUDGE-BENCH","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4203c0dc81afe24e","entry":"extract_answer","repo":"dmg-illc/JUDGE-BENCH","repo_kind":"official","path":"eval_responses.py","file_url":"https://github.com/dmg-illc/JUDGE-BENCH/blob/HEAD/eval_responses.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4203c0dc81afe24e"}},{"code_sha256_prefix":"7122e3e5a28a2808","entry":"get_files_with_responses","repo":"dmg-illc/JUDGE-BENCH","repo_kind":"official","path":"eval_responses.py","file_url":"https://github.com/dmg-illc/JUDGE-BENCH/blob/HEAD/eval_responses.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7122e3e5a28a2808"}},{"code_sha256_prefix":"98efcb39d6a74289","entry":"replace_instance","repo":"dmg-illc/JUDGE-BENCH","repo_kind":"official","path":"prompt_model_json.py","file_url":"https://github.com/dmg-illc/JUDGE-BENCH/blob/HEAD/prompt_model_json.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98efcb39d6a74289"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}