{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/collaborative-evaluation-exploring-the","title":"Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks","arxiv_id":"2310.19740","date":"2023-10-30","proceeding":null,"authors":["Qintong Li","Leyang Cui","Lingpeng Kong","Wei Bi"],"abstract":"Previous work adopts large language models (LLMs) as evaluators to evaluate natural language process (NLP) tasks. However, certain shortcomings, e.g., fairness, scope, and accuracy, persist for current LLM evaluators. To analyze whether LLMs can serve as reliable alternatives to humans, we examine the fine-grained alignment between LLM evaluators and human annotators, particularly in understanding the target evaluation tasks and conducting evaluations that meet diverse criteria. This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning), each with different evaluation criteria. Our analysis shows that 1) LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts. 2) LLM evaluators excel in general criteria, such as fluency, but face challenges with complex criteria, such as numerical reasoning. We also find that LLM-pre-drafting before human evaluation can help reduce the impact of human subjectivity and minimize annotation outliers in pure human evaluation, leading to more objective evaluation.","url_abs":"https://arxiv.org/abs/2310.19740v2","url_pdf":"https://arxiv.org/pdf/2310.19740v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"collaborative-evaluation-exploring-the","repo_url":"https://github.com/qtli/coeval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"math","task_name":"Math"},{"task_slug":"story-generation","task_name":"Story Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19740","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19740"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qtli/coeval","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bdd3c4ef3723890a","entry":"apply_rotary_pos_emb","repo":"qtli/coeval","repo_kind":"official","path":"webapp/monkey_patch_non_inplace.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/monkey_patch_non_inplace.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd3c4ef3723890a"}},{"code_sha256_prefix":"1660aaf605a919cd","entry":"firstletterupper","repo":"qtli/coeval","repo_kind":"official","path":"webapp/evaluator.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1660aaf605a919cd"}},{"code_sha256_prefix":"62a553b3b2dcaca7","entry":"rotate_half","repo":"qtli/coeval","repo_kind":"official","path":"webapp/monkey_patch_non_inplace.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/monkey_patch_non_inplace.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62a553b3b2dcaca7"}},{"code_sha256_prefix":"2cd3e6b1d3d305ca","entry":"build_logger","repo":"qtli/coeval","repo_kind":"official","path":"webapp/utils.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2cd3e6b1d3d305ca"}},{"code_sha256_prefix":"80dde39da1ba3708","entry":"compress","repo":"qtli/coeval","repo_kind":"official","path":"webapp/compression.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80dde39da1ba3708"}},{"code_sha256_prefix":"a2b79de775636d17","entry":"decompress","repo":"qtli/coeval","repo_kind":"official","path":"webapp/compression.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/compression.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2b79de775636d17"}},{"code_sha256_prefix":"2cbe6ed055d5fd44","entry":"forward","repo":"qtli/coeval","repo_kind":"official","path":"webapp/monkey_patch_non_inplace.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/monkey_patch_non_inplace.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2cbe6ed055d5fd44"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"qtli/coeval","repo_kind":"official","path":"webapp/utils.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"qtli/coeval","repo_kind":"official","path":"webapp/utils.py","file_url":"https://github.com/qtli/coeval/blob/HEAD/webapp/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}