{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/incorporating-visual-layout-structures-for","title":"VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups","arxiv_id":"2106.00676","date":"2021-06-01","proceeding":null,"authors":["Zejiang Shen","Kyle Lo","Lucy Lu Wang","Bailey Kuehl","Daniel S. Weld","Doug Downey"],"abstract":"Accurately extracting structured content from PDFs is a critical first step for NLP over scientific papers. Recent work has improved extraction accuracy by incorporating elementary layout information, e.g., each token's 2D position on the page, into language model pretraining. We introduce new methods that explicitly model VIsual LAyout (VILA) groups, i.e., text lines or text blocks, to further improve performance. In our I-VILA approach, we show that simply inserting special tokens denoting layout group boundaries into model inputs can lead to a 1.9% Macro F1 improvement in token classification. In the H-VILA approach, we show that hierarchical encoding of layout-groups can result in up-to 47% inference time reduction with less than 0.8% Macro F1 loss. Unlike prior layout-aware approaches, our methods do not require expensive additional pretraining, only fine-tuning, which we show can reduce training cost by up to 95%. Experiments are conducted on a newly curated evaluation suite, S2-VLUE, that unifies existing automatically-labeled datasets and includes a new dataset of manual annotations covering diverse papers from 19 scientific disciplines. Pre-trained weights, benchmark datasets, and source code are available at https://github.com/allenai/VILA.","url_abs":"https://arxiv.org/abs/2106.00676v3","url_pdf":"https://arxiv.org/pdf/2106.00676v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"incorporating-visual-layout-structures-for","repo_url":"https://github.com/allenai/VILA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"token-classification","task_name":"Token Classification"},{"task_slug":"token-classification","task_name":"token-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.00676","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.00676"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/VILA","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a4bbce8fc073bbb9","entry":"columns_used_in_model_inputs","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/predictors.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/predictors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4bbce8fc073bbb9"}},{"code_sha256_prefix":"8a6aeb8c178b2bb1","entry":"flatten_group_level_prediction","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/predictors.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/predictors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a6aeb8c178b2bb1"}},{"code_sha256_prefix":"3490b2bdd7f4ec6b","entry":"instantiate_textline_encoder","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/models/hierarchical_model.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/models/hierarchical_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3490b2bdd7f4ec6b"}},{"code_sha256_prefix":"364fe0222f94071b","entry":"instantiate_textline_model","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/models/hierarchical_model.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/models/hierarchical_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"364fe0222f94071b"}},{"code_sha256_prefix":"1ad2bb03b5ef4f13","entry":"is_in","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/utils.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ad2bb03b5ef4f13"}},{"code_sha256_prefix":"ef0bcc11085b5fe0","entry":"union_box","repo":"allenai/VILA","repo_kind":"official","path":"src/vila/utils.py","file_url":"https://github.com/allenai/VILA/blob/HEAD/src/vila/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef0bcc11085b5fe0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}