{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bertifying-the-hidden-markov-model-for-multi","title":"BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition","arxiv_id":"2105.12848","date":"2021-05-26","proceeding":"ACL 2021 5","authors":["Yinghao Li","Pranav Shetty","Lucas Liu","Chao Zhang","Le Song"],"abstract":"We study the problem of learning a named entity recognition (NER) tagger using noisy labels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, inaccurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hidden Markov model (CHMM), which can effectively infer true labels from multi-source noisy labels in an unsupervised way. CHMM enhances the classic hidden Markov model with the contextual representation power of pre-trained language models. Specifically, CHMM learns token-wise transition and emission probabilities from the BERT embeddings of the input tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM-ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT-NER's output is regarded as an additional weak source to train the CHMM in return. Experiments on four NER benchmarks from various domains show that our method outperforms state-of-the-art weakly supervised NER models by wide margins.","url_abs":"https://arxiv.org/abs/2105.12848v2","url_pdf":"https://arxiv.org/pdf/2105.12848v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bertifying-the-hidden-markov-model-for-multi","repo_url":"https://github.com/Yinghao-Li/CHMM-ALT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"bertifying-the-hidden-markov-model-for-multi","repo_url":"https://github.com/Yinghao-Li/Sparse-CHMM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"weakly-supervised-named-entity-recognition","task_name":"Weakly-Supervised Named Entity Recognition"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.12848","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.12848"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yinghao-Li/Sparse-CHMM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Yinghao-Li/CHMM-ALT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":20},"by_repo_kind":{"official":{"samples":15,"ran":0,"repositories":1},"listed":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"796e527adb2ed3d8","entry":"batch_prep","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"src/chmm/dataset.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/src/chmm/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"796e527adb2ed3d8"}},{"code_sha256_prefix":"6a3a0bf29f78b0d6","entry":"batch_prep","repo":"Yinghao-Li/Sparse-CHMM","repo_kind":"listed","path":"sparse_chmm/collate.py","file_url":"https://github.com/Yinghao-Li/Sparse-CHMM/blob/HEAD/sparse_chmm/collate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a3a0bf29f78b0d6"}},{"code_sha256_prefix":"ebe8e4be2bc36479","entry":"bc5cdr_annotators","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/wiser_annotator.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/wiser_annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebe8e4be2bc36479"}},{"code_sha256_prefix":"07c3eac7eb4ccba4","entry":"binary_search","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/data.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07c3eac7eb4ccba4"}},{"code_sha256_prefix":"0e2242a9ea204327","entry":"collate_fn","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"src/chmm/dataset.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/src/chmm/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e2242a9ea204327"}},{"code_sha256_prefix":"e14737726145e981","entry":"collate_fn","repo":"Yinghao-Li/Sparse-CHMM","repo_kind":"listed","path":"sparse_chmm/collate.py","file_url":"https://github.com/Yinghao-Li/Sparse-CHMM/blob/HEAD/sparse_chmm/collate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e14737726145e981"}},{"code_sha256_prefix":"1513a7e3c40ee51a","entry":"construct_spacy_doc","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/skweak_annotator.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/skweak_annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1513a7e3c40ee51a"}},{"code_sha256_prefix":"0bbaea3681a4d767","entry":"first_nonzero_idx","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/util.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bbaea3681a4d767"}},{"code_sha256_prefix":"93b0b65911d51415","entry":"get_overlaps","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/data.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93b0b65911d51415"}},{"code_sha256_prefix":"b7f01e56a9691559","entry":"laptop_annotators","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/wiser_annotator.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/wiser_annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7f01e56a9691559"}},{"code_sha256_prefix":"4182f6dbfbb21083","entry":"load_bc5cdr_sentences","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/io.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4182f6dbfbb21083"}},{"code_sha256_prefix":"80f211d366ececf3","entry":"load_conll_2003_data","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/io.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80f211d366ececf3"}},{"code_sha256_prefix":"f837fba980069ba0","entry":"load_src_metrics","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"src/chmm/dataset.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/src/chmm/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f837fba980069ba0"}},{"code_sha256_prefix":"223f33cec8c8003d","entry":"load_wikigold_data","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/io.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"223f33cec8c8003d"}},{"code_sha256_prefix":"294c159dee8bc557","entry":"log_matmul","repo":"Yinghao-Li/Sparse-CHMM","repo_kind":"listed","path":"sparse_chmm/math.py","file_url":"https://github.com/Yinghao-Li/Sparse-CHMM/blob/HEAD/sparse_chmm/math.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"294c159dee8bc557"}},{"code_sha256_prefix":"e89483f11a57dcbe","entry":"log_maxmul","repo":"Yinghao-Li/Sparse-CHMM","repo_kind":"listed","path":"sparse_chmm/math.py","file_url":"https://github.com/Yinghao-Li/Sparse-CHMM/blob/HEAD/sparse_chmm/math.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e89483f11a57dcbe"}},{"code_sha256_prefix":"9b68df96cc5a0174","entry":"logsumexp","repo":"Yinghao-Li/Sparse-CHMM","repo_kind":"listed","path":"sparse_chmm/math.py","file_url":"https://github.com/Yinghao-Li/Sparse-CHMM/blob/HEAD/sparse_chmm/math.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b68df96cc5a0174"}},{"code_sha256_prefix":"092662782020e348","entry":"ncbi_annotators","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/wiser_annotator.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/wiser_annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"092662782020e348"}},{"code_sha256_prefix":"94f946467c015b2d","entry":"one_hot","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/util.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94f946467c015b2d"}},{"code_sha256_prefix":"ceaaa5aea0a1ffec","entry":"one_hot_to_string","repo":"Yinghao-Li/CHMM-ALT","repo_kind":"official","path":"data_constr/src/util.py","file_url":"https://github.com/Yinghao-Li/CHMM-ALT/blob/HEAD/data_constr/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ceaaa5aea0a1ffec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}