{"url":"/dataset/ec-funsd","name":"EC-FUNSD","full_name":null,"description_markdown":"EC-FUNSD is introduced in  [[arXiv:2402.02379]](https://arxiv.org/abs/2402.02379) as a benchmark of semantic entity recognition (SER) and entity linking (EL), designed for the entity-centric robustness evaluation of pre-trained text-and-layout models (PTLMs).\r\n\r\nIn practical applications of document intelligence, PTLMs (e.g. the LayoutLM series) generally serve as the encoder of document layouts, similar to the role played by pre-trained contextualized language models (e.g. the BERT series) in NLP tasks.\r\nConventionally, information extraction (IE) ability of PTLMs is evaluated via SER and EL, esp. in a sequence-labeling manner. \r\nThe performance of PTLMs on these tasks reflects the capacity of their layout embeddings to facilitate downstream IE tasks.\r\nHowever, the prevailing benchmarks do not fully conform to the aforementioned evaluation pipeline, thereby diminishing the reliability of the assessment. Take FUNSD as an example, its block-level annotation falsely couples the annotations of segment and entity, which does not adequately represent semantic-driven entities and hinder the fair evaluation. \r\n\r\nThe propose of EC-FUNSD aims to provide a fair and unbiased evaluation benchmark of IE ability of PTLMs. \r\nThe construction of this dataset includes the revision of layout and IE annotations from FUNSD. \r\nFirst, the original layout annotation of FUNSD is cleaned, and multiple-row blocks are split into row-wise segments.\r\nSecond, the semantic entities are re-annotated together with their linking relationships, with the segment order preserved to ensure that each entity is represented as a continuous word span within layout, making the form of this dataset suitable for sequence-labeling models. \r\nThe final dataset consists of 199 document samples including the image, layout annotation of segments and words, and labeled entities of 3 categories. \r\nFor the detailed annotation process and statistics, please refer to the original paper.","description_withheld":null,"homepage":"https://github.com/chongzhangFDU/ROOR-Datasets","introduced_date":"2024-02-04","introduced_date_note":null,"introduced_by":{"paper":"/paper/rethinking-the-evaluation-of-pre-trained-text","title":"Rethinking the Evaluation of Pre-trained Text-and-Layout Models from an Entity-Centric Perspective","first_author":"Chong Zhang","url":null},"license":{"name":"CC-BY-4.0","url":"https://github.com/chongzhangFDU/ROOR-Datasets/blob/main/LICENSE"},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Entity Linking","url":"/task/entity-linking","datasets_with_task":"/datasets/task/entity-linking"},{"name":"Semantic entity labeling","url":"/task/semantic-entity-labeling","datasets_with_task":"/datasets/task/semantic-entity-labeling"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["EC-FUNSD"],"data_loaders":[],"num_papers_in_archive":3,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/entity-linking-on-ec-funsd","task":"Entity Linking","dataset_variant":"EC-FUNSD","rows":8,"metrics":["F1"],"first_row_in_archive_order":{"model":"RORE (GeoLayoutLM)","paper":"/paper/modeling-layout-reading-order-as-ordering","metrics":{"F1":"87.42"},"code_links":[{"title":"chongzhangFDU/ROOR","url":"https://github.com/chongzhangFDU/ROOR"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/semantic-entity-labeling-on-ec-funsd","task":"Semantic entity labeling","dataset_variant":"EC-FUNSD","rows":8,"metrics":["F1"],"first_row_in_archive_order":{"model":"RORE (LayoutLMv3-large)","paper":"/paper/modeling-layout-reading-order-as-ordering","metrics":{"F1":"84.53"},"code_links":[{"title":"chongzhangFDU/ROOR","url":"https://github.com/chongzhangFDU/ROOR"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/modeling-layout-reading-order-as-ordering","title":"Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding","date":"2024-09-29","rows_on_this_dataset":6,"code_links":1,"syntology":null},{"paper":"/paper/rethinking-the-evaluation-of-pre-trained-text","title":"Rethinking the Evaluation of Pre-trained Text-and-Layout Models from an Entity-Centric Perspective","date":"2024-02-04","rows_on_this_dataset":6,"code_links":1,"syntology":null},{"paper":"/paper/layoutlmv3-pre-training-for-document-ai-with","title":"LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking","date":"2022-04-18","rows_on_this_dataset":4,"code_links":4,"syntology":null}],"syntology_totals":{"read_at":"2026-09-25T09:33:49+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}