Browse State-of-the-Art › nlg evaluation
nlg evaluation
31 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28
Evaluate the generated text by NLG (Natural Language Generation) systems, like large language models
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 71 · with a code link: 31 · where Syntology ran a sample: 8 (5 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (8 of 71 tagged: 5 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 71 of 71, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
15 May 2023 7 repositories listedOur proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations…
-
29 Mar 2023 3 repositories listed Syntology official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 1 pointer-only (licence)In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs.
-
19 Feb 2024 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified (of 8 harvested samples)Some prior work has shown that LLMs perform well in NLG evaluation for different tasks.
-
24 May 2023 2 repositories listedMost research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements.
-
13 Oct 2022 2 repositories listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions.
-
14 Mar 2025 1 repository listedLarge Language Models (LLMs) have demonstrated great potential as evaluators of NLG systems, allowing for high-quality, reference-free, and multi-aspect assessments.
-
22 Oct 2024 1 repository listedFurthermore, we propose three perspectives that reflect the capability of meta-evaluation: discriminative power, ranking consistency, and sensitivity to score granularity.
-
26 Jun 2024 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area.
-
12 Jun 2024 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming.
-
12 Jun 2024 1 repository listedHuman evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems.
-
23 May 2024 1 repository listedSpecifically, inspired by the recent success of large language models (LLMs) in text generation and evaluation, we adopt strong LLMs as both the data generator and gold evaluator.
-
16 May 2024 1 repository listedIn this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate.
-
18 Feb 2024 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)Evaluation of opinion summaries using conventional reference-based metrics rarely provides a holistic evaluation and has been shown to have a relatively low correlation with human judgments.
-
13 Jan 2024 1 repository listedIn the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e. g., coherence, creativity, and…
-
9 Jan 2024 1 repository listedThe evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text.
-
6 Aug 2023 1 repository listedTo address this issue, we propose to utilize multiple references to enhance the consistency between these metrics and human evaluations.
-
15 Jul 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)Current developments in large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks.
-
13 Jul 2023 1 repository listedExisting evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
-
24 May 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics.
-
7 Mar 2023 1 repository listedHuman speakers can generate descriptions of perceptual concepts, abstracted from the instance-level.
-
7 Mar 2023 1 repository listedIn detail, we regard ChatGPT as a human evaluator and give task-specific (e. g., summarization) and aspect-specific (e. g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models.
-
4 Nov 2022 1 repository listedUsing the CLSE's entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian…
-
10 Oct 2022 1 repository listedIs it possible to build a general and automatic natural language generation (NLG) evaluation metric?
-
20 Sep 2022 1 repository listedIn this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics.
-
13 May 2022 1 repository listedPrecisely assessing the progress in natural language generation (NLG) tasks is challenging, and human evaluation to establish a preference in a model's output over another is often necessary.
-
16 Apr 2022 1 repository listedFor multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), e. g. mBART, the self-supervised pretraining task is trained on a wide range of monolingual languages, e. g.
-
11 Mar 2022 1 repository listedIn this work, we introduce Active Evaluation, a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms.
-
14 Sep 2021 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples)Based on the nature of information change from input to output, we classify NLG tasks into compression (e. g., summarization), transduction (e. g., text rewriting), and creation (e. g., dialog).
-
13 Sep 2021 1 repository listedNatural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e. g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc.
-
15 Mar 2021 1 repository listedAs transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations.
-
21 Jul 2017 1 repository listed
-
Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts29 Apr 2025 0 repositories listed
-
DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?10 Apr 2025 0 repositories listed
-
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators6 Mar 2025 0 repositories listed
-
SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text25 Nov 2024 0 repositories listed
-
Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking8 Nov 2024 0 repositories listed
-
Large Language Models Are Active Critics in NLG Evaluation14 Oct 2024 0 repositories listed
-
DHP Benchmark: Are LLMs Good NLG Evaluators?25 Aug 2024 0 repositories listed
-
ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models16 Jul 2024 0 repositories listed
-
WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models28 Mar 2024 0 repositories listed
-
LLM-based NLG Evaluation: Current Status and Challenges2 Feb 2024 0 repositories listed
-
The Pitfalls of Defining Hallucination15 Jan 2024 0 repositories listed
-
CoAScore: Chain-of-Aspects Prompting for NLG Evaluation16 Dec 2023 0 repositories listed
-
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects15 Nov 2023 0 repositories listed
-
Rethinking Model Evaluation as Narrowing the Socio-Technical Gap1 Jun 2023 0 repositories listed
-
Dolphin: A Challenging and Diverse Benchmark for Arabic NLG24 May 2023 0 repositories listed
-
Dialect-robust Evaluation of Generated Text2 Nov 2022 0 repositories listed
-
NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation1 Oct 2022 0 repositories listed
-
A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating1 Jul 2022 0 repositories listed
-
A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating24 May 2022 0 repositories listed
-
The Authenticity Gap in Human Evaluation24 May 2022 0 repositories listed
-
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications13 May 2022 0 repositories listed
-
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text14 Feb 2022 0 repositories listed
-
A Survey of Natural Language Generation22 Dec 2021 0 repositories listed
-
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation17 Dec 2021 0 repositories listed
-
Language Model Augmented Relevance Score19 Aug 2021 0 repositories listed
-
All That's `Human' Is Not Gold: Evaluating Human Evaluation of Generated Text1 Aug 2021 0 repositories listed
-
MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation24 Jul 2021 0 repositories listed
-
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text30 Jun 2021 0 repositories listed
-
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons16 Jun 2021 0 repositories listed
-
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation10 Jun 2021 0 repositories listed
-
A Tutorial on Evaluation Metrics used in Natural Language Generation1 Jun 2021 0 repositories listed
-
Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation1 Dec 2020 0 repositories listed
-
A Survey of Evaluation Metrics Used for NLG Systems27 Aug 2020 0 repositories listed
-
Evaluation of Text Generation: A Survey26 Jun 2020 0 repositories listed
-
Agreement is overrated: A plea for correlation to assess human evaluation reliability1 Oct 2019 0 repositories listed
-
The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations1 Oct 2019 0 repositories listed
-
Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts1 Nov 2018 0 repositories listed
-
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation29 Mar 2017 0 repositories listed
-
A Snapshot of NLG Evaluation Practices 2005 - 20141 Sep 2015 0 repositories listed
Syntology lines on 10 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections