Papers › FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation

FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation

25 Oct 2023Findings of EMNLP 2023 10archive 2025-07-28

Kun Zhang, Oana Balalau, Ioana Manolescu

Graph-to-text (G2T) generation takes a graph as input and aims to generate a fluent and faith- ful textual representation of the information in the graph. The task has many applications, such as dialogue generation and question an- swering. In this work, we investigate to what extent the G2T generation problem is solved for previously studied datasets, and how pro- posed metrics perform when comparing generated texts. To help address their limitations, we propose a new metric that correctly identifies factual faithfulness, i.e., given a triple (subject, predicate, object), it decides if the triple is present in a generated text. We show that our metric FactSpotter achieves the highest correlation with human annotations on data correct- ness, data coverage, and relevance. In addition, FactSpotter can be used as a plug-in feature to improve the factual faithfulness of existing models. Finally, we investigate if existing G2T datasets are still challenging for state-of-the-art models. Our code is available online: https://github.com/guihuzhang/FactSpotter.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

guihuzhang/FactSpotter mentioned in paperpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ACL Anthology id 2023.findings-emnlp.672 (that paper's own run record: 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Data-to-Text GenerationDialogue GenerationKG-to-Text GenerationQuestion GenerationTable-to-Text GenerationText Generation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Data-to-Text Generation DART T5B Baseline BLEU 48.47 #1 of 5 Archive leaderboard report
Data-to-Text Generation DART T5B Baseline FactSpotter 96.65 #1 of 5 Archive leaderboard report
Data-to-Text Generation DART T5B Baseline METEOR 40.74 #1 of 5 Archive leaderboard report
Data-to-Text Generation DART FactT5B BLEU 48.37 #2 of 5 Archive leaderboard report
Data-to-Text Generation DART FactT5B FactSpotter 97.60 #2 of 5 Archive leaderboard report
Data-to-Text Generation DART FactT5B METEOR 40.72 #2 of 5 Archive leaderboard report
Data-to-Text Generation DART JointGT Baseline BLEU 47.51 #4 of 5 Archive leaderboard report
Data-to-Text Generation DART JointGT Baseline FactSpotter 95.86 #4 of 5 Archive leaderboard report
Data-to-Text Generation DART JointGT Baseline METEOR 40.43 #4 of 5 Archive leaderboard report
Data-to-Text Generation DART FactJointGT BLEU 47.39 #5 of 5 Archive leaderboard report
Data-to-Text Generation DART FactJointGT FactSpotter 97.25 #5 of 5 Archive leaderboard report
Data-to-Text Generation DART FactJointGT METEOR 40.32 #5 of 5 Archive leaderboard report
Data-to-Text Generation WebNLG JointGT Baseline BLEU 67.08 #3 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG JointGT Baseline FactSpotter 99.09 #3 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG JointGT Baseline METEOR 48.34 #3 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG T5B Baseline BLEU 67.04 #4 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG T5B Baseline FactSpotter 99.44 #4 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG T5B Baseline METEOR 48.35 #4 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactT5B BLEU 67.04 #5 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactT5B FactSpotter 99.71 #5 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactT5B METEOR 48.22 #5 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactJointGT BLEU 66.89 #6 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactJointGT FactSpotter 99.67 #6 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG FactJointGT METEOR 48.19 #6 of 20 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactT5B BLEU 67.08 #1 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactT5B FactSpotter 99.71 #1 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactT5B METEOR 48.34 #1 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) JointGT Baseline BLEU 67.08 #2 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) JointGT Baseline FactSpotter 99.44 #2 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) JointGT Baseline METEOR 48.34 #2 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) T5B Baseline BLEU 67.04 #3 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) T5B Baseline FactSpotter 99.05 #3 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) T5B Baseline METEOR 48.35 #3 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactJointGT BLEU 66.89 #4 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactJointGT FactSpotter 99.67 #4 of 9 Archive leaderboard report
KG-to-Text Generation WebNLG 2.0 (Constrained) FactJointGT METEOR 48.19 #4 of 9 Archive leaderboard report
Question Generation GrailQA-Compositional FactJointGT BLEU 31.25 #1 of 4 Archive leaderboard report
Question Generation GrailQA-Compositional FactJointGT FactSpotter 97.10 #1 of 4 Archive leaderboard report
Question Generation GrailQA-Compositional FactJointGT METEOR 36.21 #1 of 4 Archive leaderboard report
Question Generation GrailQA-Compositional FactT5B BLEU 31.66 #3 of 4 Archive leaderboard report
Question Generation GrailQA-Compositional FactT5B FactSpotter 96.53 #3 of 4 Archive leaderboard report
Question Generation GrailQA-Compositional FactT5B METEOR 35.72 #3 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactT5B BLEU 46.10 #1 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactT5B FactSpotter 99.50 #1 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactT5B METEOR 42.67 #1 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactJointGT BLEU 43.61 #4 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactJointGT FactSpotter 99.12 #4 of 4 Archive leaderboard report
Question Generation GrailQA-IID FactJointGT METEOR 41.65 #4 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactJointGT FactSpotter 94.95 #2 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactJointGT METEOR 37.55 #2 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactJointGT bleu 32.46 #2 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactT5B FactSpotter 95.61 #3 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactT5B METEOR 37.39 #3 of 4 Archive leaderboard report
Question Generation GrailQA-Zero-Shot FactT5B bleu 32.39 #3 of 4 Archive leaderboard report
Table-to-Text Generation DART T5B Baseline BERT 0.9505 #1 of 6 Archive leaderboard report
Table-to-Text Generation DART T5B Baseline BLEU 48.47 #1 of 6 Archive leaderboard report
Table-to-Text Generation DART T5B Baseline BLEURT 0.6749 #1 of 6 Archive leaderboard report
Table-to-Text Generation DART T5B Baseline FactSpotter 96.65 #1 of 6 Archive leaderboard report
Table-to-Text Generation DART T5B Baseline METEOR 0.4074 #1 of 6 Archive leaderboard report
Table-to-Text Generation DART FactT5B BERT 0.9505 #2 of 6 Archive leaderboard report
Table-to-Text Generation DART FactT5B BLEU 48.37 #2 of 6 Archive leaderboard report
Table-to-Text Generation DART FactT5B BLEURT 0.6743 #2 of 6 Archive leaderboard report
Table-to-Text Generation DART FactT5B FactSpotter 97.60 #2 of 6 Archive leaderboard report
Table-to-Text Generation DART FactT5B METEOR 0.4072 #2 of 6 Archive leaderboard report
Table-to-Text Generation DART JointGT Baseline BERT 0.9492 #3 of 6 Archive leaderboard report
Table-to-Text Generation DART JointGT Baseline BLEU 47.51 #3 of 6 Archive leaderboard report
Table-to-Text Generation DART JointGT Baseline BLEURT 0.6733 #3 of 6 Archive leaderboard report
Table-to-Text Generation DART JointGT Baseline FactSpotter 95.86 #3 of 6 Archive leaderboard report
Table-to-Text Generation DART JointGT Baseline METEOR 0.4043 #3 of 6 Archive leaderboard report
Table-to-Text Generation DART FactJointGT BERT 0.9492 #4 of 6 Archive leaderboard report
Table-to-Text Generation DART FactJointGT BLEU 47.39 #4 of 6 Archive leaderboard report
Table-to-Text Generation DART FactJointGT BLEURT 0.6726 #4 of 6 Archive leaderboard report
Table-to-Text Generation DART FactJointGT FactSpotter 97.25 #4 of 6 Archive leaderboard report
Table-to-Text Generation DART FactJointGT METEOR 0.4032 #4 of 6 Archive leaderboard report
Text Generation DART T5B Baseline BLEU 48.74 #1 of 7 Archive leaderboard report
Text Generation DART T5B Baseline FactSpotter 96.65 #1 of 7 Archive leaderboard report
Text Generation DART T5B Baseline METEOR 0.4074 #1 of 7 Archive leaderboard report
Text Generation DART FactT5B BLEU 48.37 #2 of 7 Archive leaderboard report
Text Generation DART FactT5B FactSpotter 97.60 #2 of 7 Archive leaderboard report
Text Generation DART FactT5B METEOR 0.4072 #2 of 7 Archive leaderboard report
Text Generation DART JointGT Baseline BLEU 47.51 #3 of 7 Archive leaderboard report
Text Generation DART JointGT Baseline FactSpotter 0.9586 #3 of 7 Archive leaderboard report
Text Generation DART JointGT Baseline METEOR 0.4043 #3 of 7 Archive leaderboard report
Text Generation DART FactJointGT BLEU 47.39 #4 of 7 Archive leaderboard report
Text Generation DART FactJointGT FactSpotter 97.25 #4 of 7 Archive leaderboard report
Text Generation DART FactJointGT METEOR 0.4032 #4 of 7 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections