Papers › Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

18 Feb 2021arXiv:2102.09550archive 2025-07-28

Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, Gabriela Pałka

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

uakarsh/TiLT-Implementation mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Document Image Classification Visual Question Answering Visual Question Answering (VQA)document understanding

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Document Image Classification	RVL-CDIP	TILT-Large	Accuracy	95.52%	#7 of 31	Archive leaderboard	report
Document Image Classification	RVL-CDIP	TILT-Base	Accuracy	95.25%	#11 of 31	Archive leaderboard	report
Visual Question Answering (VQA)	DocVQA test	TILT-Large	ANLS	0.8705	#13 of 33	Archive leaderboard	report
Visual Question Answering (VQA)	DocVQA test	TILT-Base	ANLS	0.8392	#18 of 33	Archive leaderboard	report
Visual Question Answering (VQA)	InfographicVQA	TILT-Large	ANLS	61.20	#7 of 21	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Concatenated Skip Connection Convolution Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Max Pooling Multi-Head Attention Position-Wise Feed-Forward Layer ReLU Residual Connection Softmax Transformer U-Net

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections