Browse State-of-the-Art › Dialogue Evaluation
Dialogue Evaluation
61 papers with code · 2 benchmarks · 8 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| USR-TopicalChat (6 rows) | MDD-Eval | MDD-Eval: Self-Training on Augmented Data for Multi-Domain... | code | — | Compare |
| USR-PersonaChat (5 rows) | Lin-Reg (all) | Proxy Indicators for the Quality of Open-domain Dialogues | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
8 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 97 · with a code link: 61 · where Syntology ran a sample: 10 (8 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 97 tagged: 8 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 97 of 97, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
23 Jan 2017 8 repositories listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from…
-
18 Dec 2022 3 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches.
-
25 Oct 2022 2 repositories listedRecent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment.
-
3 Nov 2021 2 repositories listedThe development of Open-Domain Dialogue Systems (ODS)is a trending topic due to the large number of research challenges, large societal and business impact, and advances in the underlying technology.
-
23 Jun 2020 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research.
-
4 Nov 2019 2 repositories listedIn this paper, we investigate the possibility and efficacy of estimating utterance-level engagement and define a novel metric, predictive engagement, for automatic evaluation of open-domain dialogue systems.
-
24 Jul 2019 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi-reference evaluation.
-
21 Jun 2019 2 repositories listedTo investigate the strengths of this novel metric and interactive evaluation in comparison to state-of-the-art metrics and human evaluation of static conversations, we perform extended experiments with a set of models…
-
28 May 2025 1 repository listedAs the capabilities of chatbots and their underlying LLMs continue to dramatically improve, evaluating their performance has increasingly become a major blocker to their further development.
-
22 Apr 2025 1 repository listedIn this paper, we describe our participation in the RuTermEval competition devoted to extracting nested terms.
-
9 Apr 2025 1 repository listedThe participants experimented mainly with large language models in zero-shot, few-shot and fine-tuning formats.
-
7 Apr 2025 1 repository listedRecent advancements in large language models (LLMs) have revolutionized their ability to handle single-turn tasks, yet real-world applications demand sophisticated multi-turn interactions.
-
17 Jan 2025 1 repository listedOne such auxiliary loss function is Bag-of-Words (BoW) loss, defined as the cross-entropy loss for predicting all the words/tokens of the next utterance.
-
12 Jan 2025 1 repository listedAdvancements in dialogue systems powered by large language models (LLMs) have outpaced the development of reliable evaluation metrics, particularly for diverse and creative responses.
-
20 Aug 2024 1 repository listedAlthough human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also extended to dialogue.
-
16 Jul 2024 1 repository listedMotivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence).
-
24 May 2024 1 repository listedOur approach introduces several techniques: (1) Contrastive learning to differentiate between robust and non-robust response embeddings; (2) A novel metric for semantic sensitivity that combines embedding cosine…
-
1 Apr 2024 1 repository listedRecent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories.
-
1 Apr 2024 1 repository listedTrainable evaluation metrics, typically trained with true positive and randomly selected negative responses, tend to assign higher scores to responses that share greater content similarity with a given context.
-
24 Dec 2023 1 repository listedYet, existing works on utilizing LLMs for automatic dialogue evaluation are limited in their scope in terms of the number of meta-evaluation datasets, mode of evaluation, coverage of LLMs, etc.
-
3 Nov 2023 1 repository listedIn this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have.
-
13 Oct 2023 1 repository listedThe English dialogue data are extended to nine other languages with commercial machine translation systems.
-
14 Sep 2023 1 repository listedHuman evaluation has been widely accepted as the standard for evaluating chat-oriented dialogue systems.
-
31 Aug 2023 1 repository listedDespite significant research effort in the development of automatic dialogue evaluation metrics, little thought is given to evaluating dialogues other than in English.
-
31 Aug 2023 1 repository listedThe main limiting factor in the development of robust multilingual dialogue evaluation metrics is the lack of multilingual data and the limited availability of open sourced multilingual dialogue systems.
-
27 Jun 2023 1 repository listedExisting reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system.
-
8 May 2023 1 repository listedDespite the recent advances in open-domain dialogue systems, building a reliable evaluation metric is still a challenging problem.
-
28 Feb 2023 1 repository listedWe present GLM-Dialog, a large-scale language model (LLM) with 10B parameters capable of knowledge-grounded conversation in Chinese using a search engine to access the Internet knowledge.
-
17 Aug 2022 1 repository listedThis paper introduces a novel Self-supervised Fine-grained Dialogue Evaluation framework (SelF-Eval).
-
3 Jun 2022 1 repository listedThe first task is framed as a binary classification problem.
-
25 May 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (of 2 harvested samples)
-
23 May 2022 1 repository listed
-
25 Mar 2022 1 repository listed
-
18 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
11 Mar 2022 1 repository listed
-
14 Dec 2021 1 repository listed
-
1 Nov 2021 1 repository listed
-
1 Nov 2021 1 repository listed
-
1 Aug 2021 1 repository listed
-
19 Jun 2021 1 repository listed
-
10 Jun 2021 1 repository listed
-
7 Jun 2021 1 repository listed
-
5 Jun 2021 1 repository listed
-
4 Jun 2021 1 repository listed Syntology official (archive's flag): 2 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 9 harvested samples) · 2 pointer-only (licence)
-
2 Jun 2021 1 repository listed
-
1 Jun 2021 1 repository listed
-
1 Jun 2021 1 repository listed
-
6 May 2021 1 repository listed
-
16 Apr 2021 1 repository listed
-
1 Nov 2020 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples)
-
8 Oct 2020 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 3 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
23 Sep 2020 1 repository listed
-
1 Jul 2020 1 repository listed
-
1 May 2020 1 repository listed
-
1 May 2020 1 repository listed
-
6 Apr 2020 1 repository listed
-
23 Sep 2019 1 repository listed
-
6 Apr 2019 1 repository listed Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 1 pointer-only (licence)
-
23 Aug 2017 1 repository listed
-
11 Jan 2017 1 repository listed
-
DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation4 Jun 2025 0 repositories listed
-
MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation27 May 2025 0 repositories listed
-
LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation26 May 2025 0 repositories listed
-
Dialogue You Can Trust: Human and AI Perspectives on Generated Conversations3 Sep 2024 0 repositories listed
-
On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation4 Jul 2024 0 repositories listed
-
Leveraging LLMs for Dialogue Quality Measurement25 Jun 2024 0 repositories listed
-
LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation5 Jun 2024 0 repositories listed
-
CodingTeachLLM: Empowering LLM's Coding Ability via AST Prior Knowledge13 Mar 2024 0 repositories listed
-
How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation23 May 2023 0 repositories listed
-
U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendation5 May 2023 0 repositories listed
-
Pragmatically Appropriate Diversity for Dialogue Evaluation6 Apr 2023 0 repositories listed
-
Improving Open-Domain Dialogue Evaluation with a Causal Inference Model31 Jan 2023 0 repositories listed
-
PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment18 Dec 2022 0 repositories listed
-
Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations1 Oct 2022 0 repositories listed
-
Dialogue Evaluation with Offline Reinforcement Learning2 Sep 2022 0 repositories listed
-
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis1 Jul 2022 0 repositories listed
-
MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue19 Jun 2022 0 repositories listed
-
AdaCoach: A Virtual Coach for Training Customer Service Agents27 Apr 2022 0 repositories listed
-
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges18 Mar 2022 0 repositories listed
-
FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows14 Feb 2022 0 repositories listed
-
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents12 Jan 2022 0 repositories listed
-
User Response and Sentiment Prediction for Automatic Dialogue Evaluation16 Nov 2021 0 repositories listed
-
Investigating the Impact of Pre-trained Language Models on Dialog Evaluation5 Oct 2021 0 repositories listed
-
Achieving Reliable Human Assessment of Open-Domain Dialogue Systems17 Sep 2021 0 repositories listed
-
Enhancing the Open-Domain Dialogue Evaluation in Latent Space1 Aug 2021 0 repositories listed
-
DCH-2: A Parallel Customer-Helpdesk Dialogue Corpus with Distributions of Annotators' Labels18 Apr 2021 0 repositories listed
-
WeChat AI & ICT's Submission for DSTC9 Interactive Dialogue Evaluation Track20 Jan 2021 0 repositories listed
-
Learning the Human Judgment for the Automatic Evaluation of Chatbot1 May 2020 0 repositories listed
-
Predicting Ratings of Real Dialogue Participants from Artificial Data and Ratings of Human Dialogue Observers1 May 2020 0 repositories listed
-
Treating Dialogue Quality Evaluation as an Anomaly Detection Problem1 May 2020 0 repositories listed
-
How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning10 Dec 2019 0 repositories listed
-
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons6 Sep 2019 0 repositories listed
-
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings24 Apr 2019 0 repositories listed
-
Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses23 Feb 2019 0 repositories listed
-
One "Ruler" for All Languages: Multi-Lingual Dialogue Evaluation with Adversarial Multi-Task Learning8 May 2018 0 repositories listed
Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections