nlg evaluation

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 31 papers with code (71 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
  15 May 2023 7 repositories listed
  Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations…
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
  29 Mar 2023 3 repositories listed Syntology ran 2 of 4 samples · 2 unverified
  In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs.
• Are LLM-based Evaluators Confusing NLG Quality Criteria?
  19 Feb 2024 2 repositories listed Syntology ran 3 of 8 samples · 5 unverified
  Some prior work has shown that LLMs perform well in NLG evaluation for different tasks.
• Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References
  24 May 2023 2 repositories listed
  Most research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements.
• Towards a Unified Multi-Dimensional Evaluator for Text Generation
  13 Oct 2022 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions.
• OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMs
  14 Mar 2025 1 repository listed
  Large Language Models (LLMs) have demonstrated great potential as evaluators of NLG systems, allowing for high-quality, reference-free, and multi-aspect assessments.
• Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation
  22 Oct 2024 1 repository listed
  Furthermore, we propose three perspectives that reflect the capability of meta-evaluation: discriminative power, ranking consistency, and sensitivity to score granularity.
• Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
  26 Jun 2024 1 repository listed Syntology ran 5 of 5 samples · 0 unverified
  The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area.
• Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
  12 Jun 2024 1 repository listed Syntology ran 0 of 1 samples · 1 unverified · 1 pointer-only (licence)
  Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming.
• Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation
  12 Jun 2024 1 repository listed
  Human evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems.
• Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
  23 May 2024 1 repository listed
  Specifically, inspired by the recent success of large language models (LLMs) in text generation and evaluation, we adopt strong LLMs as both the data generator and gold evaluator.
• DEBATE: Devil's Advocate-Based Assessment and Text Evaluation
  16 May 2024 1 repository listed
  In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate.
• One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation
  18 Feb 2024 1 repository listed Syntology ran 5 of 5 samples · 0 unverified
  Evaluation of opinion summaries using conventional reference-based metrics rarely provides a holistic evaluation and has been shown to have a relatively low correlation with human judgments.
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
  13 Jan 2024 1 repository listed
  In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.
• LUNA: A Framework for Language Understanding and Naturalness Assessment
  9 Jan 2024 1 repository listed
  The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text.
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation
  6 Aug 2023 1 repository listed
  To address this issue, we propose to utilize \textit{multiple references} to enhance the consistency between these metrics and human evaluations.
• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models
  15 Jul 2023 1 repository listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  Current developments in large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks.
• DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
  13 Jul 2023 1 repository listed
  Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
• Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
  24 May 2023 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics.
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
  7 Mar 2023 1 repository listed
  In detail, we regard ChatGPT as a human evaluator and give task-specific (e.
• Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions
  7 Mar 2023 1 repository listed
  Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level.
• CLSE: Corpus of Linguistically Significant Entities
  4 Nov 2022 1 repository listed
  Using the CLSE's entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian…
• Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis
  10 Oct 2022 1 repository listed
  Is it possible to build a general and automatic natural language generation (NLG) evaluation metric?
• EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics
  20 Sep 2022 1 repository listed
  In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics.
• Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets
  13 May 2022 1 repository listed
  Precisely assessing the progress in natural language generation (NLG) tasks is challenging, and human evaluation to establish a preference in a model's output over another is often necessary.
• Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding
  16 Apr 2022 1 repository listed
  For multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), e.
• Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons
  11 Mar 2022 1 repository listed
  In this work, we introduce Active Evaluation, a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms.
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
  14 Sep 2021 1 repository listed Syntology ran 0 of 5 samples · 5 unverified
  Based on the nature of information change from input to output, we classify NLG tasks into compression (e.
• Perturbation CheckLists for Evaluating NLG Evaluation Metrics
  13 Sep 2021 1 repository listed
  Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.
• A Study of Automatic Metrics for the Evaluation of Natural Language Explanations
  15 Mar 2021 1 repository listed
  As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations.

Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.