{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-12247","title":"ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction","arxiv_id":"2602.12247","date":"2026-02-12","proceeding":null,"authors":["Nick Ferguson","Josh Pennington","Narek Beghian","Aravind Mohan","Douwe Kiela","Sheshansh Agrawal","Thien Hang Nguyen"],"abstract":"Unstructured documents like PDFs contain valuable structured information, but downstream systems require this data in reliable, standardized formats. LLMs are increasingly deployed to automate this extraction, making accuracy and reliability paramount. However, progress is bottlenecked by two gaps. First, no end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth. Second, no principled methodology captures the semantics of nested extraction, where fields demand different notions of correctness (exact match for identifiers, tolerance for quantities, semantic equivalence for names), arrays require alignment, and omission must be distinguished from hallucination. We address both gaps with ExtractBench, an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction. The benchmark pairs 35 PDF documents with JSON Schemas and human-annotated gold labels across economically valuable domains, yielding 12,867 evaluatable fields spanning schema complexities from tens to hundreds of fields. The evaluation framework treats the schema as an executable specification: each field declares its scoring metric. Baseline evaluations reveal that frontier models (GPT-5/5.2, Gemini-3 Flash/Pro, Claude 4.5 Opus/Sonnet) remain unreliable on realistic schemas. Performance degrades sharply with schema breadth, culminating in 0% valid output on a 369-field financial reporting schema across all tested models. We release ExtractBench at https://github.com/ContextualAI/extract-bench.","url_abs":"https://arxiv.org/abs/2602.12247","url_pdf":"https://arxiv.org/pdf/2602.12247","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.12247","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.12247"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ContextualAI/extract-bench","reach":null}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16ece47ab0eb6080","entry":"BasePrimitiveSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16ece47ab0eb6080"}},{"code_sha256_prefix":"1c7dc677df2bdb98","entry":"BooleanSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c7dc677df2bdb98"}},{"code_sha256_prefix":"c9f24e305e8f95dd","entry":"IntegerSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9f24e305e8f95dd"}},{"code_sha256_prefix":"592293abe45b17d2","entry":"ReferenceSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"592293abe45b17d2"}},{"code_sha256_prefix":"d9de76185dce3498","entry":"StringSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9de76185dce3498"}},{"code_sha256_prefix":"f3468a29bc62d363","entry":"_recursively_strip_none_from_dict","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3468a29bc62d363"}},{"code_sha256_prefix":"0bb24188d186215e","entry":"AnyOfSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bb24188d186215e"}},{"code_sha256_prefix":"3e9d1fb2e7bcfdd1","entry":"ArraySchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9d1fb2e7bcfdd1"}},{"code_sha256_prefix":"2233ec750650349c","entry":"AsyncVisitorProtocol","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2233ec750650349c"}},{"code_sha256_prefix":"32170f006a6943a3","entry":"NullSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32170f006a6943a3"}},{"code_sha256_prefix":"46afc21e27b0243e","entry":"NumberSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46afc21e27b0243e"}},{"code_sha256_prefix":"a934acc44508fb95","entry":"RootSchema","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a934acc44508fb95"}},{"code_sha256_prefix":"2ab55cac3a063283","entry":"SchemaVisitor","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ab55cac3a063283"}},{"code_sha256_prefix":"f5596648a5e69c1c","entry":"VisitorProtocol","repo":"ContextualAI/extract-bench","repo_kind":"found_in_text","path":"src/extract_bench/infra/visitors.py","file_url":"https://github.com/ContextualAI/extract-bench/blob/HEAD/src/extract_bench/infra/visitors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5596648a5e69c1c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}