Browse State-of-the-Art › Coherence Evaluation
Coherence Evaluation
15 papers with code · 2 benchmarks · 1 dataset archive 2025-07-28
Evaluating the overall coherence of text as measured by its readability and flow through ideas.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 1 newer paper was judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): GCDC + RST - Accuracy (1 judged, 0 placed), GCDC + RST - F1 (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| GCDC + RST - Accuracy (4 rows) | MTL with Transformer | Transformer Models for Text Coherence Assessment | code | — | Compare |
| GCDC + RST - F1 (3 rows) | RST-Ensemble | Neural RST-based Evaluation of Discourse Coherence | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
1 dataset whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 28 · with a code link: 15 · where Syntology ran a sample: 1 (1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (1 of 28 tagged: 1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 28 of 28, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from.
-
5 Sep 2021 2 repositories listedCoherence is an important aspect of text quality and is crucial for ensuring its readability.
-
16 Jul 2024 1 repository listedMotivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence).
-
31 Mar 2024 1 repository listedDue to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models.
-
15 Feb 2024 1 repository listedRecent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks.
-
18 Dec 2023 1 repository listedProviding natural language explanations for recommendations is particularly useful from the perspective of a non-expert user.
-
20 Jun 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We investigate CDM for open-domain text generation evaluation under two paradigms: 1) _Generative_ CDM, which harnesses the contrast of two language models' distributions to generate synthetic examples for training…
-
4 Feb 2023 1 repository listedExperiments also show that the generated captions are more coherent than that of baselines according to caption entity scores, caption Rouge scores, the two proposed coherence evaluation metrics, and human evaluations.
-
25 May 2022 1 repository listedDecision making theories such as Fuzzy-Trace Theory (FTT) suggest that individuals tend to rely on gist, or bottom-line meaning, in the text when making decisions.
-
19 May 2022 1 repository listedIn this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
-
18 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM.
-
1 Nov 2021 1 repository listedEntity grids and entity graphs are two frameworks for modeling local coherence.
-
1 Jun 2021 1 repository listedTo address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating…
-
7 May 2021 1 repository listedCoherent discourse is distinguished from a mere collection of utterances by the satisfaction of a diverse set of constraints, for example choice of expression, logical relation between denoted events, and implicit…
-
30 Sep 2020 1 repository listedWe evaluate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark.
-
14 May 2018 1 repository listedTo date there has been very little work on assessing discourse coherence methods on real-world data.
-
Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues12 Dec 2024 0 repositories listedDiscovering customer intentions in dialogue conversations is crucial for automated service agents.
-
On the Sequence Evaluation based on Stochastic Processes28 May 2024 0 repositories listedGenerative models have gained significant prominence in Natural Language Processing (NLP), especially in tackling the complex task of modeling and evaluating long text sequences.
-
BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence28 Dec 2023 0 repositories listedMeasuring the coherence of text is a vital aspect of evaluating the quality of written content.
-
Improving the Generalization Ability in Essay Coherence Evaluation through Monotonic Constraints25 Jul 2023 0 repositories listedCoherence is a crucial aspect of evaluating text readability and can be assessed through two primary factors when evaluating an essay in a scoring scenario.
-
20 Jan 2023 0 repositories listedCurrent work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them.
-
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling14 Oct 2021 0 repositories listedGiven the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models…
-
Method of the coherence evaluation of Ukrainian text31 Oct 2020 0 repositories listedDue to the growing role of the SEO technologies, it is necessary to perform an automated analysis of the article's quality.
-
On the Creation of a Corpus for Coherence Evaluation of Discursive Units1 May 2020 0 repositories listedIn this paper, we report on our experiments towards the creation of a corpus for coherence evaluation.
-
Which Evaluations Uncover Sense Representations that Actually Make Sense?1 May 2020 0 repositories listedText representations are critical for modern natural language processing.
-
Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks30 Apr 2020 0 repositories listedAlthough coherence modeling has come a long way in developing novel models, their evaluation on downstream applications for which they are purportedly developed has largely been neglected.
-
An entity-driven recursive neural network model for chinese discourse coherence modeling14 Apr 2017 0 repositories listedChinese discourse coherence modeling remains a challenge taskin Natural Language Processing field.Existing approaches mostlyfocus on the need for feature engineering, whichadoptthe sophisticated features to capture the…
-
The Sensitivity of Topic Coherence Evaluation to Topic Cardinality1 Jun 2016 0 repositories listed
-
Enriching entity grids and graphs with discourse relations: the impact in local coherence evaluation1 Nov 2015 0 repositories listed
Syntology lines on 2 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections