{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-a-unified-multi-dimensional-evaluator","title":"Towards a Unified Multi-Dimensional Evaluator for Text Generation","arxiv_id":"2210.07197","date":"2022-10-13","proceeding":null,"authors":["Ming Zhong","Yang Liu","Da Yin","Yuning Mao","Yizhu Jiao","PengFei Liu","Chenguang Zhu","Heng Ji","Jiawei Han"],"abstract":"Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository (https://github.com/maszhongming/UniEval).","url_abs":"https://arxiv.org/abs/2210.07197v1","url_pdf":"https://arxiv.org/pdf/2210.07197v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-a-unified-multi-dimensional-evaluator","repo_url":"https://github.com/maszhongming/unieval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"towards-a-unified-multi-dimensional-evaluator","repo_url":"https://github.com/hkust-knowcomp/knowledge-constrained-decoding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"response-generation","task_name":"Response Generation"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"nlg-evaluation","task_name":"nlg evaluation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2210.07197","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07197"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hkust-knowcomp/knowledge-constrained-decoding","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maszhongming/unieval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0756fa5b0d59a114","entry":"UniEvaluator","repo":"maszhongming/unieval","repo_kind":"official","path":"metric/scorer.py","file_url":"https://github.com/maszhongming/unieval/blob/HEAD/metric/scorer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0756fa5b0d59a114"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}