Papers › Improving Biomedical Pretrained Language Models with Knowledge
Improving Biomedical Pretrained Language Models with Knowledge
Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang, Fei Huang
Pretrained language models have shown success in many natural language processing tasks. Many works explore incorporating knowledge into language models. In the biomedical domain, experts have taken decades of effort on building large-scale knowledge bases. For example, the Unified Medical Language System (UMLS) contains millions of entities with their synonyms and defines hundreds of relations among entities. Leveraging this knowledge can benefit a variety of downstream tasks such as named entity recognition and relation extraction. To this end, we propose KeBioLM, a biomedical pretrained language model that explicitly leverages knowledge from the UMLS knowledge bases. Specifically, we extract entities from PubMed abstracts and link them to UMLS. We then train a knowledge-aware language model that firstly applies a text-only encoding layer to learn entity representation and applies a text-entity fusion encoding to aggregate entity representation. Besides, we add two training objectives as entity detection and entity linking. Experiments on the named entity recognition and relation extraction from the BLURB benchmark demonstrate the effectiveness of our approach. Further analysis on a collected probing dataset shows that our model has better ability to model medical knowledge.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Named Entity Recognition (NER) | BC2GM | KeBioLM | F1 | 85.1 | #9 of 13 | Archive leaderboard | report |
| Named Entity Recognition (NER) | BC5CDR-chemical | KeBioLM | F1 | 93.3 | #11 of 13 | Archive leaderboard | report |
| Named Entity Recognition (NER) | BC5CDR-disease | KeBioLM | F1 | 86.1 | #6 of 10 | Archive leaderboard | report |
| Named Entity Recognition (NER) | JNLPBA | KeBioLM | F1 | 82.0 | #1 of 17 | Archive leaderboard | report |
| Named Entity Recognition (NER) | NCBI-disease | KeBioLM | F1 | 89.1 | #6 of 26 | Archive leaderboard | report |
| Relation Extraction | ChemProt | KeBioLM | F1 | 77.5 | #5 of 13 | Archive leaderboard | report |
| Relation Extraction | DDI | KeBioLM | F1 | 81.9 | #2 of 3 | Archive leaderboard | report |
| Relation Extraction | GAD | KeBioLM | F1 | 84.3 | #2 of 3 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections