{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bros-a-layout-aware-pre-trained-language","title":"BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents","arxiv_id":"2108.04539","date":"2021-08-10","proceeding":null,"authors":["Teakgyu Hong","Donghyun Kim","Mingi Ji","Wonseok Hwang","Daehyun Nam","Sungrae Park"],"abstract":"Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods. Code is available at https://github.com/clovaai/bros.","url_abs":"https://arxiv.org/abs/2108.04539v5","url_pdf":"https://arxiv.org/pdf/2108.04539v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bros-a-layout-aware-pre-trained-language","repo_url":"https://github.com/clovaai/bros","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"bros-a-layout-aware-pre-trained-language","repo_url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/bros","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"key-information-extraction","task_name":"Key Information Extraction"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/relation-extraction-on-funsd","task":"Relation Extraction","dataset":"FUNSD","model":"BROS","rank_in_archive_order":7,"of":9,"metrics":{"F1":"77.01"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.04539","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.04539"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/bros","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clovaai/bros","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"clovaai/bros","repo_kind":"official","path":"bros/tokenization_bros.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/bros/tokenization_bros.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"clovaai/bros","repo_kind":"official","path":"bros/tokenization_bros.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/bros/tokenization_bros.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"3944c09a44634e37","entry":"do_eval_step","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_bies_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_bies_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3944c09a44634e37"}},{"code_sha256_prefix":"b0682ddacb993e88","entry":"do_eval_step","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_bio_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_bio_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0682ddacb993e88"}},{"code_sha256_prefix":"a19d5a58a6632b6a","entry":"do_eval_step","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a19d5a58a6632b6a"}},{"code_sha256_prefix":"a24cbc913ce1517c","entry":"do_eval_step","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_rel_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_rel_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a24cbc913ce1517c"}},{"code_sha256_prefix":"62ed443571cf13ef","entry":"eval_ee_bio_batch","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_bio_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_bio_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62ed443571cf13ef"}},{"code_sha256_prefix":"1bfd47528da7b59e","entry":"eval_ee_bio_example","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_bio_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_bio_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bfd47528da7b59e"}},{"code_sha256_prefix":"0e5e7de5c6b14651","entry":"eval_ee_spade_batch","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e5e7de5c6b14651"}},{"code_sha256_prefix":"cf68c6ab08d0c012","entry":"eval_ee_spade_example","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf68c6ab08d0c012"}},{"code_sha256_prefix":"ffb90147092f9680","entry":"eval_el_spade_batch","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_rel_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_rel_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffb90147092f9680"}},{"code_sha256_prefix":"8e888f85f35bcb89","entry":"eval_el_spade_example","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_spade_rel_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_spade_rel_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e888f85f35bcb89"}},{"code_sha256_prefix":"07564a0cc69b377b","entry":"get_label_map","repo":"clovaai/bros","repo_kind":"official","path":"lightning_modules/bros_bies_module.py","file_url":"https://github.com/clovaai/bros/blob/HEAD/lightning_modules/bros_bies_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07564a0cc69b377b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}