{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-bounding-box-is-worth-one-token","title":"A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding","arxiv_id":"2407.01976","date":"2024-07-02","proceeding":null,"authors":["Jinghui Lu","Haiyang Yu","Yanjie Wang","YongJie Ye","Jingqun Tang","Ziwei Yang","Binghong Wu","Qi Liu","Hao Feng","Han Wang","Hao liu","Can Huang"],"abstract":"Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at https://github.com/LayTextLLM/LayTextLLM.","url_abs":"https://arxiv.org/abs/2407.01976v3","url_pdf":"https://arxiv.org/pdf/2407.01976v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-bounding-box-is-worth-one-token","repo_url":"https://github.com/laytextllm/laytextllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"key-information-extraction","task_name":"Key Information Extraction"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"document-understanding","task_name":"document understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.01976","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01976"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/laytextllm/laytextllm","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_honours":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"d738681ea543c467","entry":"collate_fn","repo":"laytextllm/laytextllm","repo_kind":"official","path":"infer/laytextllm_inference_ds.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/infer/laytextllm_inference_ds.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d738681ea543c467"}},{"code_sha256_prefix":"e82f4d776da04670","entry":"collate_fn","repo":"laytextllm/laytextllm","repo_kind":"official","path":"train/laytextllm_train.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/train/laytextllm_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e82f4d776da04670"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"laytextllm/laytextllm","repo_kind":"official","path":"infer/laytextllm_inference.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/infer/laytextllm_inference.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"15713876351e02ec","entry":"evaluate_exact_match_accuracy","repo":"laytextllm/laytextllm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15713876351e02ec"}},{"code_sha256_prefix":"4ffcad52354b4806","entry":"evaluate_relaxed_accuracy","repo":"laytextllm/laytextllm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ffcad52354b4806"}},{"code_sha256_prefix":"d25632b12d6e4670","entry":"format_ocr_result","repo":"laytextllm/laytextllm","repo_kind":"official","path":"datasets/laytextllm_construct_example.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/datasets/laytextllm_construct_example.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d25632b12d6e4670"}},{"code_sha256_prefix":"b3c8680ebd8739af","entry":"relaxed_correctness","repo":"laytextllm/laytextllm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/laytextllm/laytextllm/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3c8680ebd8739af"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}