{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rocketeval-efficient-automated-llm-evaluation","title":"RocketEval: Efficient Automated LLM Evaluation via Grading Checklist","arxiv_id":"2503.05142","date":"2025-03-07","proceeding":null,"authors":["Tianjun Wei","Wei Wen","Ruizhi Qiao","Xing Sun","Jianghong Ma"],"abstract":"Evaluating large language models (LLMs) in diverse and challenging scenarios is essential to align them with human preferences. To mitigate the prohibitive costs associated with human evaluations, utilizing a powerful LLM as a judge has emerged as a favored approach. Nevertheless, this methodology encounters several challenges, including substantial expenses, concerns regarding privacy and security, and reproducibility. In this paper, we propose a straightforward, replicable, and accurate automated evaluation method by leveraging a lightweight LLM as the judge, named RocketEval. Initially, we identify that the performance disparity between lightweight and powerful LLMs in evaluation tasks primarily stems from their ability to conduct comprehensive analyses, which is not easily enhanced through techniques such as chain-of-thought reasoning. By reframing the evaluation task as a multi-faceted Q&A using an instance-specific checklist, we demonstrate that the limited judgment accuracy of lightweight LLMs is largely attributes to high uncertainty and positional bias. To address these challenges, we introduce an automated evaluation process grounded in checklist grading, which is designed to accommodate a variety of scenarios and questions. This process encompasses the creation of checklists, the grading of these checklists by lightweight LLMs, and the reweighting of checklist items to align with the supervised annotations. Our experiments carried out on the automated evaluation benchmarks, MT-Bench and WildBench datasets, reveal that RocketEval, when using Gemma-2-2B as the judge, achieves a high correlation (0.965) with human preferences, which is comparable to GPT-4o. Moreover, RocketEval provides a cost reduction exceeding 50-fold for large-scale evaluation and comparison scenarios. Our code is available at https://github.com/Joinn99/RocketEval-ICLR .","url_abs":"https://arxiv.org/abs/2503.05142v1","url_pdf":"https://arxiv.org/pdf/2503.05142v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rocketeval-efficient-automated-llm-evaluation","repo_url":"https://github.com/joinn99/rocketeval-iclr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.05142","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.05142"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Joinn99/RocketEval-ICLR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joinn99/rocketeval-iclr","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b13720137bf9399f","entry":"chat_history","repo":"joinn99/rocketeval-iclr","repo_kind":"official","path":"src/rocketeval/process/checklist.py","file_url":"https://github.com/joinn99/rocketeval-iclr/blob/HEAD/src/rocketeval/process/checklist.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b13720137bf9399f"}},{"code_sha256_prefix":"1275ab0a44ccee37","entry":"load_bench_data","repo":"joinn99/rocketeval-iclr","repo_kind":"official","path":"src/rocketeval/process/checklist.py","file_url":"https://github.com/joinn99/rocketeval-iclr/blob/HEAD/src/rocketeval/process/checklist.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1275ab0a44ccee37"}},{"code_sha256_prefix":"e779fa1c3f5da509","entry":"prompt_to_message","repo":"joinn99/rocketeval-iclr","repo_kind":"official","path":"src/rocketeval/process/checklist.py","file_url":"https://github.com/joinn99/rocketeval-iclr/blob/HEAD/src/rocketeval/process/checklist.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e779fa1c3f5da509"}},{"code_sha256_prefix":"a3000858ed4771a8","entry":"row_create","repo":"joinn99/rocketeval-iclr","repo_kind":"official","path":"src/rocketeval/process/checklist.py","file_url":"https://github.com/joinn99/rocketeval-iclr/blob/HEAD/src/rocketeval/process/checklist.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3000858ed4771a8"}},{"code_sha256_prefix":"70d43bc3b7e3389f","entry":"create_checklist","repo":"joinn99/rocketeval-iclr","repo_kind":"official","path":"src/rocketeval/process/checklist.py","file_url":"https://github.com/joinn99/rocketeval-iclr/blob/HEAD/src/rocketeval/process/checklist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70d43bc3b7e3389f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}