{"url":"/dataset/llm-health-benchmarks","name":"LLM Health Benchmarks","full_name":"LLM Health Benchmarks - Yesil Science","description_markdown":"## **LLM Health Benchmarks Dataset**\r\n\r\nThe **Health Benchmarks Dataset** is a specialized resource for evaluating large language models (LLMs) in different medical specialties. It provides structured question-answer pairs designed to test the performance of AI models in understanding and generating domain-specific knowledge.\r\n\r\n\r\n### **Primary Purpose**\r\n\r\nThis dataset is built to:\r\n- Benchmark LLMs in **medical specialties and subfields**.\r\n- Assess the **accuracy** and **contextual understanding** of AI in healthcare.\r\n- Serve as a **standardized evaluation suite** for AI systems designed for medical applications.\r\n\r\n#### **Key Features**\r\n- Covers **50+ medical and health-related topics**, including both clinical and non-clinical domains.\r\n- Includes **~7,500 structured question-answer pairs**.\r\n- Designed for fine-grained performance evaluation in medical specialties.\r\n\r\n#### **Applications**\r\n- **LLM Evaluation**: Benchmarking AI models for domain-specific performance.\r\n- **Healthcare AI Research**: Standardized testing for AI in healthcare.\r\n- **Medical Education AI**: Testing AI systems designed for tutoring medical students.\r\n\r\n\r\n\r\n### **Dataset Structure**\r\n\r\nThe dataset is organized by medical specialties and subfields, each represented as a split. Below is a snapshot:\r\n\r\n| **Specialty**              | **Number of Rows** |\r\n|-----------------------------|--------------------|\r\n| Lab Medicine               | 158                |\r\n| Ethics                     | 174                |\r\n| Dermatology                | 170                |\r\n| Gastroenterology           | 163                |\r\n| Internal Medicine          | 178                |\r\n| Oncology                   | 180                |\r\n| Orthopedics                | 177                |\r\n| General Surgery            | 178                |\r\n| Pediatrics                 | 180                |\r\n| ...(and more)              | ...                |\r\n\r\nEach split contains:\r\n- **`Questions`**: The medical questions for the specialty.\r\n- **`Answers`**: Corresponding high-quality answers.\r\n\r\n\r\n### **Usage Instructions**\r\n\r\nHere’s how you can load and use the dataset:\r\n\r\n```python\r\nfrom datasets import load_dataset\r\n\r\n## Load the dataset\r\ndataset = load_dataset(\"yesilhealth/Health_Benchmarks\")\r\n\r\n## Access specific specialty splits\r\noncology = dataset[\"Oncology\"]\r\ninternal_medicine = dataset[\"Internal_Medicine\"]\r\n\r\n## View sample data\r\nprint(oncology[:5])\r\n```\r\n\r\n\r\n### **Evaluation Workflow**\r\n\r\n1. **Model Input**: Provide the questions from each split to the LLM.\r\n2. **Model Output**: Collect the AI-generated answers.\r\n3. **Scoring**: Compare model answers to ground truth answers using metrics such as:\r\n   - Exact Match (EM)\r\n   - F1 Score\r\n   - Semantic Similarity\r\n\r\n\r\n### **Citation**\r\n\r\nIf you use this dataset for research or development, please cite:\r\n\r\n```plaintext\r\n@dataset{yesilhealth_health_benchmarks,\r\n  title={Health Benchmarks Dataset},\r\n  author={Yesil Health AI},\r\n  year={2024},\r\n  url={https://huggingface.co/datasets/yesilhealth/Health_Benchmarks}\r\n}\r\n```\r\n\r\n\r\n### **License**\r\n\r\nThis dataset is licensed under the **[Apache 2.0 License](LICENSE)**.\r\n\r\n\r\n### **Feedback**\r\n\r\nFor questions, suggestions, or feedback, feel free to contact us via email at [hello@yesilhealth.com].","description_withheld":null,"homepage":"https://yesilscience.com/llm-health-benchmark","introduced_date":"2025-02-14","introduced_date_note":null,"introduced_by":null,"license":{"name":"Apache 2.0 License","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"},{"name":"Medical","url":"/datasets/modality/medical"}],"tasks":[{"name":"Medical Diagnosis","url":"/task/medical-diagnosis","datasets_with_task":"/datasets/task/medical-diagnosis"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["LLM Health Benchmarks"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-25T09:33:49+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}