{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-language-models-be-biomedical-knowledge","title":"Can Language Models be Biomedical Knowledge Bases?","arxiv_id":"2109.07154","date":"2021-09-15","proceeding":"EMNLP 2021 11","authors":["Mujeen Sung","Jinhyuk Lee","Sean Yi","Minji Jeon","Sungdong Kim","Jaewoo Kang"],"abstract":"Pre-trained language models (LMs) have become ubiquitous in solving various natural language processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these powerful LMs can be used as domain-specific KBs. To this end, we create the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge triples for probing biomedical LMs. We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge. Although this seems promising given the task difficulty, our detailed analyses reveal that most predictions are highly correlated with prompt templates without any subjects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BioLAMA can serve as a challenging benchmark for biomedical factual probing.","url_abs":"https://arxiv.org/abs/2109.07154v1","url_pdf":"https://arxiv.org/pdf/2109.07154v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-language-models-be-biomedical-knowledge","repo_url":"https://github.com/dmis-lab/biolama","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"biolama","name":"BioLAMA","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.07154","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.07154"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmis-lab/biolama","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/princeton-nlp/OptiPrompt","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jzbjyb/X-FACTR","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ae552334dd6723dd","entry":"convert_manual_to_dense","repo":"dmis-lab/biolama","repo_kind":"official","path":"BioLAMA/run_optiprompt.py","file_url":"https://github.com/dmis-lab/biolama/blob/HEAD/BioLAMA/run_optiprompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae552334dd6723dd"}},{"code_sha256_prefix":"bdc2968261f9093b","entry":"get_new_token","repo":"dmis-lab/biolama","repo_kind":"official","path":"BioLAMA/run_optiprompt.py","file_url":"https://github.com/dmis-lab/biolama/blob/HEAD/BioLAMA/run_optiprompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdc2968261f9093b"}},{"code_sha256_prefix":"a75dcf58b2351e5c","entry":"init_template","repo":"dmis-lab/biolama","repo_kind":"official","path":"BioLAMA/run_optiprompt.py","file_url":"https://github.com/dmis-lab/biolama/blob/HEAD/BioLAMA/run_optiprompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a75dcf58b2351e5c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}