{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ernie-layout-layout-knowledge-enhanced-pre","title":"ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding","arxiv_id":"2210.06155","date":"2022-10-12","proceeding":null,"authors":["Qiming Peng","Yinxu Pan","Wenjin Wang","Bin Luo","Zhenyu Zhang","Zhengjie Huang","Teng Hu","Weichong Yin","Yongfeng Chen","Yin Zhang","Shikun Feng","Yu Sun","Hao Tian","Hua Wu","Haifeng Wang"],"abstract":"Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge, leading to sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel document pre-training solution with layout knowledge enhancement in the whole workflow, to learn better representations that combine the features from text, layout, and image. Specifically, we first rearrange input sequences in the serialization stage, and then present a correlative pre-training task, reading order prediction, to learn the proper reading order of documents. To improve the layout awareness of the model, we integrate a spatial-aware disentangled attention into the multi-modal transformer and a replaced regions prediction task into the pre-training phase. Experimental results show that ERNIE-Layout achieves superior performance on various downstream tasks, setting new state-of-the-art on key information extraction, document image classification, and document question answering datasets. The code and models are publicly available at http://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout.","url_abs":"https://arxiv.org/abs/2210.06155v2","url_pdf":"https://arxiv.org/pdf/2210.06155v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ernie-layout-layout-knowledge-enhanced-pre","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"ernie-layout-layout-knowledge-enhanced-pre","repo_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"document-image-classification","task_name":"Document Image Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"key-information-extraction","task_name":"Key Information Extraction"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"semantic-entity-labeling","task_name":"Semantic entity labeling"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"document-understanding","task_name":"document understanding"},{"task_slug":"document-image-classification","task_name":"document-image-classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-entity-labeling-on-funsd","task":"Semantic entity labeling","dataset":"FUNSD","model":"ERNIE-Layoutlarge","rank_in_archive_order":2,"of":15,"metrics":{"F1":"93.12"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"ERNIE-Layout large (ensemble)","rank_in_archive_order":8,"of":33,"metrics":{"ANLS":"0.8841"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"ERNIE-Layout large","rank_in_archive_order":16,"of":33,"metrics":{"ANLS":"0.8486"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.06155","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06155"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NormXU/ERNIE-Layout-Pytorch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_fixture":1,"unverified":5},"by_repo_kind":{"listed":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/modeling_erine_layout_extrapolation.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/modeling_erine_layout_extrapolation.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"e62e48a00ed06f63","entry":"pad_at_dim","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/modeling_erine_layout_extrapolation.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/modeling_erine_layout_extrapolation.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e62e48a00ed06f63"}},{"code_sha256_prefix":"015a941d044dac12","entry":"ResNet101","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/visual_backbone.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/visual_backbone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"015a941d044dac12"}},{"code_sha256_prefix":"8d726fd8fa82d461","entry":"ResNet152","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/visual_backbone.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/visual_backbone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d726fd8fa82d461"}},{"code_sha256_prefix":"363d1ff55bddc72b","entry":"ResNet50","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/visual_backbone.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/visual_backbone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"363d1ff55bddc72b"}},{"code_sha256_prefix":"d4c1c406c2087434","entry":"drop_path","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/modeling_erine_layout_extrapolation.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/modeling_erine_layout_extrapolation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4c1c406c2087434"}},{"code_sha256_prefix":"bca8304ee996b24d","entry":"relative_position_bucket","repo":"NormXU/ERNIE-Layout-Pytorch","repo_kind":"listed","path":"networks/modeling_erine_layout.py","file_url":"https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/HEAD/networks/modeling_erine_layout.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bca8304ee996b24d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}