Papers › LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

29 Dec 2020ACL 2021 5arXiv:2012.14740archive 2025-07-28

Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 → 0.8420), CORD (0.9493 → 0.9601), SROIE (0.9524 → 0.9781), Kleister-NDA (0.8340 → 0.8520), RVL-CDIP (0.9443 → 0.9564), and DocVQA (0.7295 → 0.8672). We made our model and code publicly available at https://aka.ms/layoutlmv2.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

microsoft/unilm officialmentioned in papermentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (arXiv:2002.12804; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified), BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254; that paper's own run record: 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified), DeepNet: Scaling Transformers to 1,000 Layers (arXiv:2203.00555; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
facebookresearch/data2vec_vision mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254; that paper's own run record: 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified); nothing checks that they implement this paper's method. report
huggingface/transformers mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
MS-P3/code3 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to JetMoE: Reaching Llama2 Performance with 0.1M Dollars (arXiv:2404.07413; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
MindSpore-scientific-2/code-14 mindsporeSyntology: not harvested report
MindSpore-scientific/code-7 Syntology: not harvested report
PaddlePaddle/PaddleNLP paddleSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation... (arXiv:2301.03416; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
PaddlePaddle/PaddleOCR paddleApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to... (arXiv:2508.19944; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
pwc-1/Paper-9 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RandAugment: Practical automated data augmentation with a reduced search space (arXiv:1909.13719; that paper's own run record: 58 ran (of which 1 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 51 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Document Image ClassificationDocument Layout AnalysisKey Information ExtractionKey-value Pair ExtractionLanguage ModelingLanguage ModellingRelation ExtractionSemantic entity labelingVisual Question AnsweringVisual Question Answering (VQA)document understanding

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Document Image Classification RVL-CDIP LayoutLMv2LARGE Accuracy 95.64% #6 of 31 Archive leaderboard report
Document Image Classification RVL-CDIP LayoutLMv2BASE Accuracy 95.25% #12 of 31 Archive leaderboard report
Document Image Classification RVL-CDIP LayoutLMv2BASE Parameters 200M #12 of 31 Archive leaderboard report
Key Information Extraction CORD LayoutLMv2LARGE F1 96.01 #8 of 9 Archive leaderboard report
Key Information Extraction CORD LayoutLMv2BASE F1 94.95 #9 of 9 Archive leaderboard report
Key Information Extraction Kleister NDA LayoutLMv2LARGE F1 85.2 #1 of 3 Archive leaderboard report
Key Information Extraction Kleister NDA LayoutLMv2BASE F1 83.3 #2 of 3 Archive leaderboard report
Key Information Extraction SROIE LayoutLMv2LARGE (Excluding OCR mismatch) F1 97.81 #1 of 5 Archive leaderboard report
Key Information Extraction SROIE LayoutLMv2LARGE F1 96.61 #3 of 5 Archive leaderboard report
Key Information Extraction SROIE LayoutLMv2BASE F1 96.25 #4 of 5 Archive leaderboard report
Key-value Pair Extraction RFUND-EN LayoutLMv2_base key-value pair F1 49.06 #12 of 13 Archive leaderboard report
Relation Extraction FUNSD LayoutLMv2 large F1 70.57 #8 of 9 Archive leaderboard report
Semantic entity labeling FUNSD LayoutLMv2LARGE F1 84.2 #12 of 15 Archive leaderboard report
Semantic entity labeling FUNSD LayoutLMv2BASE F1 82.76 #14 of 15 Archive leaderboard report
Visual Question Answering (VQA) DocVQA test LayoutLMv2LARGE ANLS 0.8672 #15 of 33 Archive leaderboard report
Visual Question Answering (VQA) DocVQA test LayoutLMv2BASE ANLS 0.7808 #25 of 33 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

1x1 ConvolutionAbsolute Position EncodingsAdamAttentionAverage PoolingBPEBatch NormalizationConvolutionDense ConnectionsDropoutFPNGlobal Average PoolingGrouped ConvolutionKaiming InitializationLabel SmoothingLayer NormalizationLayoutLMv2Linear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerReLUResNeXtResNeXt BlockResidual ConnectionSoftmaxTransformerWordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections