{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reading-order-matters-information-extraction","title":"Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction","arxiv_id":"2310.11016","date":"2023-10-17","proceeding":null,"authors":["Chong Zhang","Ya Guo","Yi Tu","Huan Chen","Jinyang Tang","Huijia Zhu","Qi Zhang","Tao Gui"],"abstract":"Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs), in which named entity recognition (NER) is treated as a sequence-labeling task of predicting the BIO entity tags for tokens, following the typical setting of NLP. However, BIO-tagging scheme relies on the correct order of model inputs, which is not guaranteed in real-world NER on scanned VrDs where text are recognized and arranged by OCR systems. Such reading order issue hinders the accurate marking of entities by BIO-tagging scheme, making it impossible for sequence-labeling methods to predict correct named entities. To address the reading order issue, we introduce Token Path Prediction (TPP), a simple prediction head to predict entity mentions as token sequences within documents. Alternative to token classification, TPP models the document layout as a complete directed graph of tokens, and predicts token paths within the graph as entities. For better evaluation of VrD-NER systems, we also propose two revised benchmark datasets of NER on scanned documents which can reflect real-world scenarios. Experiment results demonstrate the effectiveness of our method, and suggest its potential to be a universal solution to various information extraction tasks on documents.","url_abs":"https://arxiv.org/abs/2310.11016v1","url_pdf":"https://arxiv.org/pdf/2310.11016v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reading-order-matters-information-extraction","repo_url":"https://github.com/chongzhangfdu/tpp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"reading-order-matters-information-extraction","repo_url":"https://github.com/WinterShiver/Token-Path-Prediction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"entity-linking","task_name":"Entity Linking"},{"task_slug":"key-information-extraction","task_name":"Key Information Extraction"},{"task_slug":"key-value-pair-extraction","task_name":"Key-value Pair Extraction"},{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"reading-order-detection","task_name":"Reading Order Detection"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"semantic-entity-labeling","task_name":"Semantic entity labeling"},{"task_slug":"sentence-ordering","task_name":"Sentence Ordering"},{"task_slug":"token-classification","task_name":"Token Classification"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"},{"task_slug":"token-classification","task_name":"token-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-linking-on-funsd","task":"Entity Linking","dataset":"FUNSD","model":"TPP (LayoutMask)","rank_in_archive_order":3,"of":7,"metrics":{"F1":"79.20"},"uses_additional_data":false},{"leaderboard":"/sota/key-information-extraction-on-cord","task":"Key Information Extraction","dataset":"CORD","model":"TPP (LayoutMask)","rank_in_archive_order":6,"of":9,"metrics":{"F1":"96.92"},"uses_additional_data":false},{"leaderboard":"/sota/key-value-pair-extraction-on-rfund-en","task":"Key-value Pair Extraction","dataset":"RFUND-EN","model":"TPP\n(LayoutLMv3_base)","rank_in_archive_order":11,"of":13,"metrics":{"key-value pair F1":"50.27"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-cord-r","task":"Named Entity Recognition (NER)","dataset":"CORD-r","model":"TPP (LayoutLMv3)","rank_in_archive_order":1,"of":4,"metrics":{"F1":"91.85"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-cord-r","task":"Named Entity Recognition (NER)","dataset":"CORD-r","model":"TPP (LayoutMask)","rank_in_archive_order":2,"of":4,"metrics":{"F1":"89.34"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-funsd-r","task":"Named Entity Recognition (NER)","dataset":"FUNSD-r","model":"TPP (LayoutLMv3)","rank_in_archive_order":1,"of":4,"metrics":{"F1":"80.40"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-funsd-r","task":"Named Entity Recognition (NER)","dataset":"FUNSD-r","model":"TPP (LayoutMask)","rank_in_archive_order":3,"of":4,"metrics":{"F1":"78.19"},"uses_additional_data":false},{"leaderboard":"/sota/reading-order-detection-on-roor","task":"Reading Order Detection","dataset":"ROOR","model":"TPP (LayoutLMv3-base)","rank_in_archive_order":3,"of":4,"metrics":{"Segment-level F1":"42.96"},"uses_additional_data":false},{"leaderboard":"/sota/reading-order-detection-on-readingbank","task":"Reading Order Detection","dataset":"ReadingBank","model":"TPP (LayoutMask)","rank_in_archive_order":1,"of":2,"metrics":{"Average Page-level BLEU":"98.16","Average Relative Distance (ARD)":"0.37"},"uses_additional_data":false},{"leaderboard":"/sota/relation-extraction-on-funsd","task":"Relation Extraction","dataset":"FUNSD","model":"TPP (LayoutMask)","rank_in_archive_order":6,"of":9,"metrics":{"F1":"79.20"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-entity-labeling-on-funsd","task":"Semantic entity labeling","dataset":"FUNSD","model":"TPP (LayoutMask)","rank_in_archive_order":11,"of":15,"metrics":{"F1":"85.16"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.11016","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}