{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/modeling-layout-reading-order-as-ordering","title":"Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding","arxiv_id":"2409.19672","date":"2024-09-29","proceeding":null,"authors":["Chong Zhang","Yi Tu","Yixi Zhao","Chenshu Yuan","Huan Chen","Yue Zhang","Mingxu Chai","Ya Guo","Huijia Zhu","Qi Zhang","Tao Gui"],"abstract":"Modeling and leveraging layout reading order in visually-rich documents (VrDs) is critical in document intelligence as it captures the rich structure semantics within documents. Previous works typically formulated layout reading order as a permutation of layout elements, i.e. a sequence containing all the layout elements. However, we argue that this formulation does not adequately convey the complete reading order information in the layout, which may potentially lead to performance decline in downstream VrD tasks. To address this issue, we propose to model the layout reading order as ordering relations over the set of layout elements, which have sufficient expressive capability for the complete reading order information. To enable empirical evaluation on methods towards the improved form of reading order prediction (ROP), we establish a comprehensive benchmark dataset including the reading order annotation as relations over layout elements, together with a relation-extraction-based method that outperforms previous methods. Moreover, to highlight the practical benefits of introducing the improved form of layout reading order, we propose a reading-order-relation-enhancing pipeline to improve model performance on any arbitrary VrD task by introducing additional reading order relation inputs. Comprehensive results demonstrate that the pipeline generally benefits downstream VrD tasks: (1) with utilizing the reading order relation information, the enhanced downstream models achieve SOTA results on both two task settings of the targeted dataset; (2) with utilizing the pseudo reading order information generated by the proposed ROP model, the performance of the enhanced models has improved across all three models and eight cross-domain VrD-IE/QA task settings without targeted optimization.","url_abs":"https://arxiv.org/abs/2409.19672v1","url_pdf":"https://arxiv.org/pdf/2409.19672v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"modeling-layout-reading-order-as-ordering","repo_url":"https://github.com/chongzhangFDU/ROOR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"entity-linking","task_name":"Entity Linking"},{"task_slug":"key-information-extraction","task_name":"Key Information Extraction"},{"task_slug":"reading-order-detection","task_name":"Reading Order Detection"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"semantic-entity-labeling","task_name":"Semantic entity labeling"},{"task_slug":"document-understanding","task_name":"document understanding"}],"methods":[],"datasets_introduced":[{"slug":"roor","name":"ROOR","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-linking-on-ec-funsd","task":"Entity Linking","dataset":"EC-FUNSD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":1,"of":8,"metrics":{"F1":"87.42"},"uses_additional_data":false},{"leaderboard":"/sota/entity-linking-on-ec-funsd","task":"Entity Linking","dataset":"EC-FUNSD","model":"RORE (LayoutLMv3-large)","rank_in_archive_order":3,"of":8,"metrics":{"F1":"79.33"},"uses_additional_data":false},{"leaderboard":"/sota/entity-linking-on-ec-funsd","task":"Entity Linking","dataset":"EC-FUNSD","model":"RORE (LayoutLMv3-base)","rank_in_archive_order":6,"of":8,"metrics":{"F1":"73.64"},"uses_additional_data":false},{"leaderboard":"/sota/entity-linking-on-funsd","task":"Entity Linking","dataset":"FUNSD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":2,"of":7,"metrics":{"F1":"88.46"},"uses_additional_data":false},{"leaderboard":"/sota/key-information-extraction-on-cord","task":"Key Information Extraction","dataset":"CORD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":1,"of":9,"metrics":{"F1":"98.52"},"uses_additional_data":false},{"leaderboard":"/sota/key-information-extraction-on-sroie","task":"Key Information Extraction","dataset":"SROIE","model":"RORE (GeoLayoutLM)","rank_in_archive_order":2,"of":5,"metrics":{"F1":"96.97"},"uses_additional_data":false},{"leaderboard":"/sota/reading-order-detection-on-roor","task":"Reading Order Detection","dataset":"ROOR","model":"LayoutLMv3-GlobalPointer (large)","rank_in_archive_order":1,"of":4,"metrics":{"Segment-level F1":"82.38"},"uses_additional_data":false},{"leaderboard":"/sota/reading-order-detection-on-roor","task":"Reading Order Detection","dataset":"ROOR","model":"LayoutLMv3-GlobalPointer (base)","rank_in_archive_order":2,"of":4,"metrics":{"Segment-level F1":"68.60"},"uses_additional_data":false},{"leaderboard":"/sota/relation-extraction-on-funsd","task":"Relation Extraction","dataset":"FUNSD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":3,"of":9,"metrics":{"F1":"88.46"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-entity-labeling-on-ec-funsd","task":"Semantic entity labeling","dataset":"EC-FUNSD","model":"RORE (LayoutLMv3-large)","rank_in_archive_order":1,"of":8,"metrics":{"F1":"84.53"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-entity-labeling-on-ec-funsd","task":"Semantic entity labeling","dataset":"EC-FUNSD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":2,"of":8,"metrics":{"F1":"84.34"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-entity-labeling-on-ec-funsd","task":"Semantic entity labeling","dataset":"EC-FUNSD","model":"RORE (LayoutLMv3-base)","rank_in_archive_order":6,"of":8,"metrics":{"F1":"82.80"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-entity-labeling-on-funsd","task":"Semantic entity labeling","dataset":"FUNSD","model":"RORE (GeoLayoutLM)","rank_in_archive_order":6,"of":15,"metrics":{"F1":"91.84"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.19672","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}