{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/systematic-evaluation-of-llm-as-a-judge-in","title":"Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates","arxiv_id":"2408.13006","date":"2024-08-23","proceeding":null,"authors":["Hui Wei","Shenghua He","Tian Xia","Fei Liu","Andy Wong","Jingyang Lin","Mei Han"],"abstract":"LLM-as-a-Judge has been widely applied to evaluate and compare different LLM alignmnet approaches (e.g., RLHF and DPO). However, concerns regarding its reliability have emerged, due to LLM judges' biases and inconsistent decision-making. Previous research has developed evaluation frameworks to assess reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address LLM internal inconsistency. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-Judge methods, leading to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM-as-a-Judge on alignment tasks by defining more theoretically interpretable evaluation metrics and explicitly mitigating LLM internal inconsistency from reliability metrics. We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges, which facilitates practitioners to choose LLM judges for alignment tasks. In the experiments, we examine effects of diverse prompt templates on LLM-judge reliability and also demonstrate our developed framework by comparing various LLM judges on two common alignment datasets (i.e., TL;DR Summarization and HH-RLHF-Helpfulness). Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.","url_abs":"https://arxiv.org/abs/2408.13006v2","url_pdf":"https://arxiv.org/pdf/2408.13006v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"systematic-evaluation-of-llm-as-a-judge-in","repo_url":"https://github.com/shenghh2015/llm-judge-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dpo","method_name":"DPO"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.13006","atlas_url":"https://app.syntology.ai/?focus=2408.13006","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.13006"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shenghh2015/llm-judge-eval","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"5025a6a8bfcb633d","entry":"create_comparison_prompt","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_judge.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5025a6a8bfcb633d"}},{"code_sha256_prefix":"12f7cbf0ec203ecb","entry":"extract_template_name","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_visualize.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12f7cbf0ec203ecb"}},{"code_sha256_prefix":"a8b1ffab6e08bd4a","entry":"find_latest","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_others.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_others.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8b1ffab6e08bd4a"}},{"code_sha256_prefix":"86106f801d5dc2c8","entry":"get_std","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_visualize.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86106f801d5dc2c8"}},{"code_sha256_prefix":"14c3351cff9920bc","entry":"import_template","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_judge.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14c3351cff9920bc"}},{"code_sha256_prefix":"e84225b86120d10a","entry":"remove_redundant","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_judge.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e84225b86120d10a"}},{"code_sha256_prefix":"ecc8c3ebd720ff32","entry":"get_mean","repo":"shenghh2015/llm-judge-eval","repo_kind":"official","path":"utils/utils_visualize.py","file_url":"https://github.com/shenghh2015/llm-judge-eval/blob/HEAD/utils/utils_visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecc8c3ebd720ff32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}