Papers › Text-to-Text Pre-Training for Data-to-Text Tasks

Text-to-Text Pre-Training for Data-to-Text Tasks

21 May 2020INLG (ACL) 2020 12arXiv:2005.10433archive 2025-07-28

Mihir Kale, Abhinav Rastogi

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

google-research-datasets/ToTTo officialmentioned in papermentioned on GitHub report

shark-nlp/cont mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Data-to-Text Generation Language Modeling Language Modelling Text Generation Transfer Learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Data-to-Text Generation	MULTIWOZ 2.1	T5-Base	BLEU	35.1	#1 of 5	Archive leaderboard	report
Data-to-Text Generation	ToTTo	T5-3B	BLEU	49.5	#1 of 6	Archive leaderboard	report
Data-to-Text Generation	ToTTo	T5-3B	PARENT	58.4	#1 of 6	Archive leaderboard	report
Data-to-Text Generation	WebNLG	T5-Base	BLEU	64.7	#11 of 20	Archive leaderboard	report
Data-to-Text Generation	WebNLG Full	T5-Large	BLEU	57.1	#5 of 8	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adafactor Attention Attention Dropout BPE Dense Connections Dropout Gated Linear Unit Inverse Square Root Schedule Layer Normalization Linear Layer Multi-Head Attention Residual Connection SentencePiece Softmax T5

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections