Browse State-of-the-Art › Coherence Evaluation

Coherence Evaluation

15 papers with code · 2 benchmarks · 1 dataset archive 2025-07-28

Natural Language Processing

Evaluating the overall coherence of text as measured by its readability and flow through ideas.

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology 1 newer paper was judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): GCDC + RST - Accuracy (1 judged, 0 placed), GCDC + RST - F1 (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

DatasetBest model (first row in archive order)PaperCodeSyntologyCompare
GCDC + RST - Accuracy (4 rows) MTL with Transformer Transformer Models for Text Coherence Assessment code — Compare
GCDC + RST - F1 (3 rows) RST-Ensemble Neural RST-based Evaluation of Discourse Coherence code — Compare

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

1 dataset whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Papers archive 2025-07-28

archive papers tagged: 28 · with a code link: 15 · where Syntology ran a sample: 1 (1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (1 of 28 tagged: 1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 28 of 28, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from.

  • 5 Sep 2021 2 repositories listed
    Coherence is an important aspect of text quality and is crucial for ensuring its readability.
  • 16 Jul 2024 1 repository listed
    Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence).
  • 31 Mar 2024 1 repository listed
    Due to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models.
  • 15 Feb 2024 1 repository listed
    Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks.
  • 18 Dec 2023 1 repository listed
    Providing natural language explanations for recommendations is particularly useful from the perspective of a non-expert user.
  • We investigate CDM for open-domain text generation evaluation under two paradigms: 1) _Generative_ CDM, which harnesses the contrast of two language models' distributions to generate synthetic examples for training…
  • 4 Feb 2023 1 repository listed
    Experiments also show that the generated captions are more coherent than that of baselines according to caption entity scores, caption Rouge scores, the two proposed coherence evaluation metrics, and human evaluations.
  • 25 May 2022 1 repository listed
    Decision making theories such as Fuzzy-Trace Theory (FTT) suggest that individuals tend to rely on gist, or bottom-line meaning, in the text when making decisions.
  • 19 May 2022 1 repository listed
    In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
  • 18 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
    We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM.
  • 1 Nov 2021 1 repository listed
    Entity grids and entity graphs are two frameworks for modeling local coherence.
  • 1 Jun 2021 1 repository listed
    To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating…
  • 7 May 2021 1 repository listed
    Coherent discourse is distinguished from a mere collection of utterances by the satisfaction of a diverse set of constraints, for example choice of expression, logical relation between denoted events, and implicit…
  • 30 Sep 2020 1 repository listed
    We evaluate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark.
  • 14 May 2018 1 repository listed
    To date there has been very little work on assessing discourse coherence methods on real-world data.
  • Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues
    12 Dec 2024 0 repositories listed
    Discovering customer intentions in dialogue conversations is crucial for automated service agents.
  • On the Sequence Evaluation based on Stochastic Processes
    28 May 2024 0 repositories listed
    Generative models have gained significant prominence in Natural Language Processing (NLP), especially in tackling the complex task of modeling and evaluating long text sequences.
  • BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence
    28 Dec 2023 0 repositories listed
    Measuring the coherence of text is a vital aspect of evaluating the quality of written content.
  • Improving the Generalization Ability in Essay Coherence Evaluation through Monotonic Constraints
    25 Jul 2023 0 repositories listed
    Coherence is a crucial aspect of evaluating text readability and can be assessed through two primary factors when evaluating an essay in a scoring scenario.
  • 20 Jan 2023 0 repositories listed
    Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them.
  • Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling
    14 Oct 2021 0 repositories listed
    Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models…
  • Method of the coherence evaluation of Ukrainian text
    31 Oct 2020 0 repositories listed
    Due to the growing role of the SEO technologies, it is necessary to perform an automated analysis of the article's quality.
  • On the Creation of a Corpus for Coherence Evaluation of Discursive Units
    1 May 2020 0 repositories listed
    In this paper, we report on our experiments towards the creation of a corpus for coherence evaluation.
  • Which Evaluations Uncover Sense Representations that Actually Make Sense?
    1 May 2020 0 repositories listed
    Text representations are critical for modern natural language processing.
  • Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks
    30 Apr 2020 0 repositories listed
    Although coherence modeling has come a long way in developing novel models, their evaluation on downstream applications for which they are purportedly developed has largely been neglected.
  • An entity-driven recursive neural network model for chinese discourse coherence modeling
    14 Apr 2017 0 repositories listed
    Chinese discourse coherence modeling remains a challenge taskin Natural Language Processing field.Existing approaches mostlyfocus on the need for feature engineering, whichadoptthe sophisticated features to capture the…
  • The Sensitivity of Topic Coherence Evaluation to Topic Cardinality
    1 Jun 2016 0 repositories listed
  • Enriching entity grids and graphs with discourse relations: the impact in local coherence evaluation
    1 Nov 2015 0 repositories listed

Syntology lines on 2 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections