Papers › Text-to-Text Pre-Training for Data-to-Text Tasks

Text-to-Text Pre-Training for Data-to-Text Tasks

21 May 2020INLG (ACL) 2020 12arXiv:2005.10433archive 2025-07-28

Mihir Kale, Abhinav Rastogi

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

google-research-datasets/ToTTo officialmentioned in papermentioned on GitHubSyntology: not harvested report
shark-nlp/cont mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Data-to-Text GenerationLanguage ModelingLanguage ModellingText GenerationTransfer Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Data-to-Text Generation MULTIWOZ 2.1 T5-Base BLEU 35.1 #1 of 5 Archive leaderboard report
Data-to-Text Generation ToTTo T5-3B BLEU 49.5 #1 of 6 Archive leaderboard report
Data-to-Text Generation ToTTo T5-3B PARENT 58.4 #1 of 6 Archive leaderboard report
Data-to-Text Generation WebNLG T5-Base BLEU 64.7 #11 of 20 Archive leaderboard report
Data-to-Text Generation WebNLG Full T5-Large BLEU 57.1 #5 of 8 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdafactorAttentionAttention DropoutBPEDense ConnectionsDropoutGated Linear UnitInverse Square Root ScheduleLayer NormalizationLinear LayerMulti-Head AttentionResidual ConnectionSentencePieceSoftmaxT5

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections