Papers › Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical...

Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding

9 Apr 2023arXiv:2304.05368archive 2025-07-28

Yuqing Wang, Yun Zhao, Linda Petzold

Large language models (LLMs) have made significant progress in various domains, including healthcare. However, the specialized nature of clinical language understanding tasks presents unique challenges and limitations that warrant further investigation. In this study, we conduct a comprehensive evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within the realm of clinical language understanding tasks. These tasks span a diverse range, including named entity recognition, relation extraction, natural language inference, semantic textual similarity, document classification, and question-answering. We also introduce a novel prompting strategy, self-questioning prompting (SQP), tailored to enhance LLMs' performance by eliciting informative questions and answers pertinent to the clinical scenarios at hand. Our evaluation underscores the significance of task-specific learning strategies and prompting techniques for improving LLMs' effectiveness in healthcare-related tasks. Additionally, our in-depth error analysis on the challenging relation extraction task offers valuable insights into error distribution and potential avenues for improvement using SQP. Our study sheds light on the practical implications of employing LLMs in the specialized domain of healthcare, serving as a foundation for future research and the development of potential applications in healthcare settings.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

eternityyw/llm_healthcare officialmentioned in papermentioned on GitHubMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Document Classification Named Entity Recognition Natural Language Inference Question Answering Relation Extraction Semantic Textual Similarity named-entity-recognition

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Absolute Position Encodings Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Dropout GPT-3 Label Smoothing Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections