{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/effective-use-of-bidirectional-language","title":"Effective Use of Bidirectional Language Modeling for Transfer Learning in Biomedical Named Entity Recognition","arxiv_id":"1711.07908","date":"2017-11-21","proceeding":null,"authors":["Devendra Singh Sachan","Pengtao Xie","Mrinmaya Sachan","Eric P. Xing"],"abstract":"Biomedical named entity recognition (NER) is a fundamental task in text\nmining of medical documents and has many applications. Deep learning based\napproaches to this task have been gaining increasing attention in recent years\nas their parameters can be learned end-to-end without the need for\nhand-engineered features. However, these approaches rely on high-quality\nlabeled data, which is expensive to obtain. To address this issue, we\ninvestigate how to use unlabeled text data to improve the performance of NER\nmodels. Specifically, we train a bidirectional language model (BiLM) on\nunlabeled data and transfer its weights to \"pretrain\" an NER model with the\nsame architecture as the BiLM, which results in a better parameter\ninitialization of the NER model. We evaluate our approach on four benchmark\ndatasets for biomedical NER and show that it leads to a substantial improvement\nin the F1 scores compared with the state-of-the-art approaches. We also show\nthat BiLM weight transfer leads to a faster model training and the pretrained\nmodel requires fewer training examples to achieve a particular F1 score.","url_abs":"http://arxiv.org/abs/1711.07908v3","url_pdf":"http://arxiv.org/pdf/1711.07908v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"effective-use-of-bidirectional-language","repo_url":"https://github.com/charles9n/bert-sklearn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"effective-use-of-bidirectional-language","repo_url":"https://github.com/sreejukomath/NLPProjects","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.07908","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}