{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bioelectra-pretrained-biomedical-text-encoder","title":"BioELECTRA:Pretrained Biomedical text Encoder using Discriminators","arxiv_id":null,"date":"2021-06-11","proceeding":"ACL Anthology 2021 6","authors":["Kamal raj Kanakarajan","Bhuvana Kundumani","Malaikannan Sankarasubbu"],"abstract":"Recent advancements in pretraining strategies in NLP have shown a significant improvement in the performance of models on various text mining tasks. We apply ‘replaced token detection’ pretraining technique proposed by ELECTRA and pretrain a biomedical language model from scratch using biomedical text and vocabulary. We introduce BioELECTRA, a biomedical domain-specific language encoder model that adapts ELECTRA for the Biomedical domain. WE evaluate our model on the BLURB and BLUE biomedical NLP benchmarks. BioELECTRA outperforms the previous models and achieves state of the art (SOTA) on all the 13 datasets in BLURB benchmark and on all the 4 Clinical datasets from BLUE Benchmark across 7 different NLP tasks. BioELECTRA pretrained on PubMed and PMC full text articles performs very well on Clinical datasets as well. BioELECTRA achieves new SOTA 86.34%(1.39% accuracy improvement) on MedNLI and 64% (2.98% accuracy improvement) on PubMedQA dataset.","url_abs":"https://aclanthology.org/2021.bionlp-1.16","url_pdf":"https://aclanthology.org/2021.bionlp-1.16.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bioelectra-pretrained-biomedical-text-encoder","repo_url":"https://github.com/kamalkraj/BioELECTRA","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"medical-named-entity-recognition","task_name":"Medical Named Entity Recognition"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence-similarity","task_name":"Sentence Similarity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"electra","method_name":"ELECTRA"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/medical-named-entity-recognition-on-share","task":"Medical Named Entity Recognition","dataset":"ShARe/CLEF eHealth corpus","model":"BioELECTRA","rank_in_archive_order":1,"of":4,"metrics":{"F1":"0.8371"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-mednli","task":"Natural Language Inference","dataset":"MedNLI","model":"BioELECTRA-Base","rank_in_archive_order":3,"of":7,"metrics":{"Accuracy":"86.34","Params (M)":"110"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-pubmedqa","task":"Question Answering","dataset":"PubMedQA","model":"BioELECTRA uncased","rank_in_archive_order":26,"of":30,"metrics":{"Accuracy":"64.2"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}