{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multifaceteval-multifaceted-evaluation-to","title":"MultifacetEval: Multifaceted Evaluation to Probe LLMs in Mastering Medical Knowledge","arxiv_id":"2406.02919","date":"2024-06-05","proceeding":null,"authors":["Yuxuan Zhou","Xien Liu","Chen Ning","Ji Wu"],"abstract":"Large language models (LLMs) have excelled across domains, also delivering notable performance on the medical evaluation benchmarks, such as MedQA. However, there still exists a significant gap between the reported performance and the practical effectiveness in real-world medical scenarios. In this paper, we aim to explore the causes of this gap by employing a multifaceted examination schema to systematically probe the actual mastery of medical knowledge by current LLMs. Specifically, we develop a novel evaluation framework MultifacetEval to examine the degree and coverage of LLMs in encoding and mastering medical knowledge at multiple facets (comparison, rectification, discrimination, and verification) concurrently. Based on the MultifacetEval framework, we construct two multifaceted evaluation datasets: MultiDiseK (by producing questions from a clinical disease knowledge base) and MultiMedQA (by rephrasing each question from a medical benchmark MedQA into multifaceted questions). The experimental results on these multifaceted datasets demonstrate that the extent of current LLMs in mastering medical knowledge is far below their performance on existing medical benchmarks, suggesting that they lack depth, precision, and comprehensiveness in mastering medical knowledge. Consequently, current LLMs are not yet ready for application in real-world medical tasks. The codes and datasets are available at https://github.com/THUMLP/MultifacetEval.","url_abs":"https://arxiv.org/abs/2406.02919v1","url_pdf":"https://arxiv.org/pdf/2406.02919v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multifaceteval-multifaceted-evaluation-to","repo_url":"https://github.com/thumlp/multifaceteval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"MedQA"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.02919","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02919"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thumlp/multifaceteval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/THUMLP/MultifacetEval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":19,"unverified":4},"by_repo_kind":{"official":{"samples":24,"ran":20,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8dadf8eedaeb7704","entry":"load_data","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gemini_medqa_ao.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gemini_medqa_ao.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dadf8eedaeb7704"}},{"code_sha256_prefix":"1914087ade7e8bca","entry":"answer_vote","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MAQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MAQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1914087ade7e8bca"}},{"code_sha256_prefix":"bad8bcc825e86b66","entry":"answer_vote","repo":"thumlp/multifaceteval","repo_kind":"official","path":"TFQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/TFQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bad8bcc825e86b66"}},{"code_sha256_prefix":"eaec6bcf4919e973","entry":"cal_relative_performance","repo":"thumlp/multifaceteval","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eaec6bcf4919e973"}},{"code_sha256_prefix":"4310ef2a5f4665b3","entry":"chating","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gemini_medqa_ao.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gemini_medqa_ao.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4310ef2a5f4665b3"}},{"code_sha256_prefix":"e0a0dd11c62a40fc","entry":"chating","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gemini_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gemini_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0a0dd11c62a40fc"}},{"code_sha256_prefix":"20ea85577d06ac6d","entry":"chating","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gpt_medqa_ao.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gpt_medqa_ao.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20ea85577d06ac6d"}},{"code_sha256_prefix":"a621bd45961d965f","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MAQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MAQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a621bd45961d965f"}},{"code_sha256_prefix":"24796f0e8267d08f","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MAQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MAQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24796f0e8267d08f"}},{"code_sha256_prefix":"5fe4ca43949d2549","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MCQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MCQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fe4ca43949d2549"}},{"code_sha256_prefix":"172aa259fd1693c0","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MCQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MCQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"172aa259fd1693c0"}},{"code_sha256_prefix":"be9589b46024780d","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be9589b46024780d"}},{"code_sha256_prefix":"66dd4af07fab516b","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66dd4af07fab516b"}},{"code_sha256_prefix":"e508780be7f14ee4","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"TFQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/TFQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e508780be7f14ee4"}},{"code_sha256_prefix":"685e445c8af90359","entry":"extract_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"TFQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/TFQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"685e445c8af90359"}},{"code_sha256_prefix":"2afd82f1ded8da3d","entry":"extract_judge_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2afd82f1ded8da3d"}},{"code_sha256_prefix":"adc04b1d9b3eac03","entry":"extract_judge_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adc04b1d9b3eac03"}},{"code_sha256_prefix":"eb38f86f64a280f1","entry":"option_vote","repo":"thumlp/multifaceteval","repo_kind":"official","path":"MAQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/MAQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb38f86f64a280f1"}},{"code_sha256_prefix":"50ad33216c34d9df","entry":"prepare_examples","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gemini_medqa_ao.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gemini_medqa_ao.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50ad33216c34d9df"}},{"code_sha256_prefix":"d1c4dd08b81436c2","entry":"prepare_examples","repo":"thumlp/multifaceteval","repo_kind":"official","path":"evaluate_gemini_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/evaluate_gemini_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1c4dd08b81436c2"}},{"code_sha256_prefix":"f1992607d65dde76","entry":"analyze_results","repo":"thumlp/multifaceteval","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1992607d65dde76"}},{"code_sha256_prefix":"362d000f1db36524","entry":"extract_option_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"362d000f1db36524"}},{"code_sha256_prefix":"c092350d57b92b7c","entry":"extract_option_ans","repo":"thumlp/multifaceteval","repo_kind":"official","path":"RQ_answer_analysis_medqa_cotsc.py","file_url":"https://github.com/thumlp/multifaceteval/blob/HEAD/RQ_answer_analysis_medqa_cotsc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c092350d57b92b7c"}},{"code_sha256_prefix":"5a00271a076bdc13","entry":"recognize_and_rewrite","repo":"THUMLP/MultifacetEval","repo_kind":"official","path":"recognize_and_rewrite_medqa.py","file_url":"https://github.com/THUMLP/MultifacetEval/blob/HEAD/recognize_and_rewrite_medqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a00271a076bdc13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}