{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/biomegatron-larger-biomedical-domain-language","title":"BioMegatron: Larger Biomedical Domain Language Model","arxiv_id":"2010.06060","date":"2020-10-12","proceeding":"EMNLP 2020 11","authors":["Hoo-chang Shin","Yang Zhang","Evelina Bakhturina","Raul Puri","Mostofa Patwary","Mohammad Shoeybi","Raghav Mani"],"abstract":"There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books. Yet, most works do not study the factors affecting each domain language application deeply. Additionally, the study of model size on domain-specific models has been mostly missing. We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer. We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications. We demonstrate noticeable improvements over the previous state-of-the-art (SOTA) on standard biomedical NLP benchmarks of named entity recognition, relation extraction, and question answering. Model checkpoints and code are available at [https://ngc.nvidia.com] and [https://github.com/NVIDIA/NeMo].","url_abs":"https://arxiv.org/abs/2010.06060v2","url_pdf":"https://arxiv.org/pdf/2010.06060v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"biomegatron-larger-biomedical-domain-language","repo_url":"https://github.com/NVIDIA/NeMo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"model","task_name":"model"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-chemical","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-chemical","model":"BioMegatron","rank_in_archive_order":12,"of":13,"metrics":{"F1":"92.9"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-disease","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-disease","model":"BioMegatron","rank_in_archive_order":1,"of":10,"metrics":{"F1":"88.5"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI-disease","model":"BioMegatron BERT-cased","rank_in_archive_order":17,"of":26,"metrics":{"F1":"87.8"},"uses_additional_data":false},{"leaderboard":"/sota/relation-extraction-on-chemprot","task":"Relation Extraction","dataset":"ChemProt","model":"BioMegatron","rank_in_archive_order":7,"of":13,"metrics":{"F1":"77.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.06060","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}