{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/k-qa-a-real-world-medical-q-a-benchmark","title":"K-QA: A Real-World Medical Q&A Benchmark","arxiv_id":"2401.14493","date":"2024-01-25","proceeding":null,"authors":["Itay Manes","Naama Ronn","David Cohen","Ran Ilan Ber","Zehavi Horowitz-Kugler","Gabriel Stanovsky"],"abstract":"Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health. To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform). We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements. Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer. Finally, we use K-QA along with these metrics to evaluate several state-of-the-art models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors. Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations. We make K-QA available to to the community to spur research into medically accurate NLP applications.","url_abs":"https://arxiv.org/abs/2401.14493v1","url_pdf":"https://arxiv.org/pdf/2401.14493v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"k-qa-a-real-world-medical-q-a-benchmark","repo_url":"https://github.com/itaymanes/k-qa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"k-qa","name":"k-qa","full_name":"K-QA: A Real-World Medical Q&A Benchmark"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.14493","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.14493"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/itaymanes/k-qa","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f33257122a3e9df1","entry":"compute_overall_metrics","repo":"itaymanes/k-qa","repo_kind":"official","path":"evaluation/metrics.py","file_url":"https://github.com/itaymanes/k-qa/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f33257122a3e9df1"}},{"code_sha256_prefix":"fa788cda714fd442","entry":"compute_precision","repo":"itaymanes/k-qa","repo_kind":"official","path":"evaluation/metrics.py","file_url":"https://github.com/itaymanes/k-qa/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa788cda714fd442"}},{"code_sha256_prefix":"faf5dd03e0b09ce3","entry":"compute_recall","repo":"itaymanes/k-qa","repo_kind":"official","path":"evaluation/metrics.py","file_url":"https://github.com/itaymanes/k-qa/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"faf5dd03e0b09ce3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}