{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/medexqa-medical-question-answering-benchmark","title":"MedExQA: Medical Question Answering Benchmark with Multiple Explanations","arxiv_id":"2406.06331","date":"2024-06-10","proceeding":null,"authors":["Yunsoo Kim","Jinge Wu","Yusuf Abdulle","Honghan Wu"],"abstract":"This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.","url_abs":"https://arxiv.org/abs/2406.06331v2","url_pdf":"https://arxiv.org/pdf/2406.06331v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"medexqa-medical-question-answering-benchmark","repo_url":"https://github.com/knowlab/medexqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"Medical Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.06331","atlas_url":"https://app.syntology.ai/?focus=2406.06331","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06331"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/knowlab/medexqa","reach":null}],"summary":{"ran_draft_wrong":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"1a8789ccdb8f2890","entry":"format_example","repo":"knowlab/medexqa","repo_kind":"official","path":"evaluate_MedExQA.py","file_url":"https://github.com/knowlab/medexqa/blob/HEAD/evaluate_MedExQA.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a8789ccdb8f2890"}},{"code_sha256_prefix":"2836f4edcd88af80","entry":"format_example","repo":"knowlab/medexqa","repo_kind":"official","path":"evaluate_pipe_MedExQA.py","file_url":"https://github.com/knowlab/medexqa/blob/HEAD/evaluate_pipe_MedExQA.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2836f4edcd88af80"}},{"code_sha256_prefix":"c6184ec494c39842","entry":"generate_few_shot_prompt","repo":"knowlab/medexqa","repo_kind":"official","path":"evaluate_MedExQA.py","file_url":"https://github.com/knowlab/medexqa/blob/HEAD/evaluate_MedExQA.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6184ec494c39842"}},{"code_sha256_prefix":"2b2c3f7907a0b36d","entry":"process_before_extraction","repo":"knowlab/medexqa","repo_kind":"official","path":"evaluate_pipe_MedExQA.py","file_url":"https://github.com/knowlab/medexqa/blob/HEAD/evaluate_pipe_MedExQA.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b2c3f7907a0b36d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}