{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-cross-sentence-contexts-for-named","title":"Exploring Cross-sentence Contexts for Named Entity Recognition with BERT","arxiv_id":"2006.01563","date":"2020-06-02","proceeding":"COLING 2020 8","authors":["Jouni Luoma","Sampo Pyysalo"],"abstract":"Named entity recognition (NER) is frequently addressed as a sequence classification task where each input consists of one sentence of text. It is nevertheless clear that useful information for the task can often be found outside of the scope of a single-sentence context. Recently proposed self-attention models such as BERT can both efficiently capture long-distance relationships in input as well as represent inputs consisting of several sentences, creating new opportunitites for approaches that incorporate cross-sentence information in natural language processing tasks. In this paper, we present a systematic study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context in the form of additional sentences to BERT input systematically increases NER performance on all of the tested languages and models. Including multiple sentences in each input also allows us to study the predictions of the same sentences in different contexts. We propose a straightforward method, Contextual Majority Voting (CMV), to combine different predictions for sentences and demonstrate this to further increase NER performance with BERT. Our approach does not require any changes to the underlying BERT architecture, rather relying on restructuring examples for training and prediction. Evaluation on established datasets, including the CoNLL'02 and CoNLL'03 NER benchmarks, demonstrates that our proposed approach can improve on the state-of-the-art NER results on English, Dutch, and Finnish, achieves the best reported BERT-based results on German, and is on par with performance reported with other BERT-based approaches in Spanish. We release all methods implemented in this work under open licenses.","url_abs":"https://arxiv.org/abs/2006.01563v2","url_pdf":"https://arxiv.org/pdf/2006.01563v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-cross-sentence-contexts-for-named","repo_url":"https://github.com/jouniluoma/bert-ner-cmv","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/named-entity-recognition-on-conll-2002-dutch","task":"Named Entity Recognition (NER)","dataset":"CoNLL 2002 (Dutch)","model":"Cross-sentence context (CMV)","rank_in_archive_order":5,"of":6,"metrics":{"F1":"93.49"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-conll-2002","task":"Named Entity Recognition (NER)","dataset":"CoNLL 2002 (Spanish)","model":"Cross-sentence context (CMV)","rank_in_archive_order":6,"of":6,"metrics":{"F1":"88.32"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-conll-2003","task":"Named Entity Recognition (NER)","dataset":"CoNLL 2003 (English)","model":"Cross-sentence context (First)","rank_in_archive_order":13,"of":73,"metrics":{"F1":"93.74"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-conll-2003-german","task":"Named Entity Recognition (NER)","dataset":"CoNLL 2003 (German)","model":"Cross-sentence context (CMV)","rank_in_archive_order":3,"of":6,"metrics":{"F1":"87.31"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.01563","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.01563"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jouniluoma/bert-ner-cmv","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1923fc05163d207d","entry":"convert_to_unicode","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"bert_tokenization.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/bert_tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1923fc05163d207d"}},{"code_sha256_prefix":"772a429073a9ffad","entry":"evaluate","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"conlleval.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/conlleval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"772a429073a9ffad"}},{"code_sha256_prefix":"30b05f2a415dd132","entry":"load_vocab","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"bert_tokenization.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/bert_tokenization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30b05f2a415dd132"}},{"code_sha256_prefix":"6dfc81a02ee3dd85","entry":"parse_args","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"conlleval.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/conlleval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6dfc81a02ee3dd85"}},{"code_sha256_prefix":"2e9480de024f3eec","entry":"parse_tag","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"conlleval.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/conlleval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e9480de024f3eec"}},{"code_sha256_prefix":"0e5615f8994003cf","entry":"printable_text","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"bert_tokenization.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/bert_tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e5615f8994003cf"}},{"code_sha256_prefix":"65943daabb32c33b","entry":"read_conll","repo":"jouniluoma/bert-ner-cmv","repo_kind":"official","path":"compare.py","file_url":"https://github.com/jouniluoma/bert-ner-cmv/blob/HEAD/compare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65943daabb32c33b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}