{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gpteval-nlg-evaluation-using-gpt-4-with","title":"G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment","arxiv_id":"2303.16634","date":"2023-03-29","proceeding":null,"authors":["Yang Liu","Dan Iter","Yichong Xu","Shuohang Wang","Ruochen Xu","Chenguang Zhu"],"abstract":"The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts. The code is at https://github.com/nlpyang/geval","url_abs":"https://arxiv.org/abs/2303.16634v3","url_pdf":"https://arxiv.org/pdf/2303.16634v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gpteval-nlg-evaluation-using-gpt-4-with","repo_url":"https://github.com/nlpyang/geval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gpteval-nlg-evaluation-using-gpt-4-with","repo_url":"https://github.com/megagonlabs/llm-longeval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"gpteval-nlg-evaluation-using-gpt-4-with","repo_url":"https://github.com/tongbao96/code-for-sfr-as","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"nlg-evaluation","task_name":"nlg evaluation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.16634","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16634"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tongbao96/code-for-sfr-as","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/nlpyang/geval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megagonlabs/llm-longeval","reach":null}],"summary":{"ran":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f8f9c3b9fc2b97e9","entry":"calculate_correlation","repo":"nlpyang/geval","repo_kind":"official","path":"meta_eval_summeval.py","file_url":"https://github.com/nlpyang/geval/blob/HEAD/meta_eval_summeval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8f9c3b9fc2b97e9"}},{"code_sha256_prefix":"f662bdaf543ab06e","entry":"parse_score","repo":"megagonlabs/llm-longeval","repo_kind":"listed","path":"eval/evaluator.py","file_url":"https://github.com/megagonlabs/llm-longeval/blob/HEAD/eval/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f662bdaf543ab06e"}},{"code_sha256_prefix":"c1344440d3eb8ce8","entry":"Evaluator","repo":"megagonlabs/llm-longeval","repo_kind":"listed","path":"eval/evaluator.py","file_url":"https://github.com/megagonlabs/llm-longeval/blob/HEAD/eval/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c1344440d3eb8ce8"}},{"code_sha256_prefix":"94622d9c122d3bb0","entry":"parse_output","repo":"nlpyang/geval","repo_kind":"official","path":"meta_eval_summeval.py","file_url":"https://github.com/nlpyang/geval/blob/HEAD/meta_eval_summeval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94622d9c122d3bb0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}