{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/moving-beyond-medical-exam-questions-a","title":"Moving Beyond Medical Exam Questions: A Clinician-Annotated Dataset of Real-World Tasks and Ambiguity in Mental Healthcare","arxiv_id":"2502.16051","date":"2025-02-22","proceeding":null,"authors":["Max Lamparth","Declan Grabb","Amy Franks","Scott Gershan","Kaitlyn N. Kunstman","Aaron Lulla","Monika Drummond Roots","Manu Sharma","Aryan Shrivastava","Nina Vasan","Colleen Waickman"],"abstract":"Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice board exam questions. Thus, we present an expert-created and annotated dataset spanning five critical domains of decision-making in mental healthcare: treatment, diagnosis, documentation, monitoring, and triage. This dataset - created without any LM assistance - is designed to capture the nuanced clinical reasoning and daily ambiguities mental health practitioners encounter, reflecting the inherent complexities of care delivery that are missing from existing datasets. Almost all 203 base questions with five answer options each have had the decision-irrelevant demographic patient information removed and replaced with variables (e.g., AGE), and are available for male, female, or non-binary-coded patients. For question categories dealing with ambiguity and multiple valid answer options, we create a preference dataset with uncertainties from the expert annotations. We outline a series of intended use cases and demonstrate the usability of our dataset by evaluating eleven off-the-shelf and four mental health fine-tuned LMs on category-specific task accuracy, on the impact of patient demographic information on decision-making, and how consistently free-form responses deviate from human annotated samples.","url_abs":"https://arxiv.org/abs/2502.16051v2","url_pdf":"https://arxiv.org/pdf/2502.16051v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"moving-beyond-medical-exam-questions-a","repo_url":"https://github.com/maxlampe/mentat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.16051","atlas_url":"https://app.syntology.ai/?focus=2502.16051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.16051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maxlampe/mentat","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12},"by_repo_kind":{"official":{"samples":12,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1c537b3738787652","entry":"acc_helper","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/analysis/analysis_helper_functions.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/analysis/analysis_helper_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c537b3738787652"}},{"code_sha256_prefix":"79acc06aee50bf59","entry":"bootstrap_wrap","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/pipeline/bootstrap_tools.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/pipeline/bootstrap_tools.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79acc06aee50bf59"}},{"code_sha256_prefix":"b8dd32c4ef929742","entry":"calculate_cross_entropy","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_llama2.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_llama2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8dd32c4ef929742"}},{"code_sha256_prefix":"e14dc3791b438d6c","entry":"calculate_cross_entropy","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_openai.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_openai.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e14dc3791b438d6c"}},{"code_sha256_prefix":"0e2db02475497ead","entry":"calculate_model_probs","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_llama2.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_llama2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e2db02475497ead"}},{"code_sha256_prefix":"8ccdc497155ee3bc","entry":"calculate_model_probs","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_openai.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_openai.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ccdc497155ee3bc"}},{"code_sha256_prefix":"75eacc9208462688","entry":"check_is_correct","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_llama2.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_llama2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75eacc9208462688"}},{"code_sha256_prefix":"6b828a6120688ec1","entry":"check_is_correct","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_llama3.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_llama3.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b828a6120688ec1"}},{"code_sha256_prefix":"a6a1ac4f0fbcedd2","entry":"convert_df_to_huggingface","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/pipeline/data_struct.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/pipeline/data_struct.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6a1ac4f0fbcedd2"}},{"code_sha256_prefix":"a20d88089985381e","entry":"convert_questions_to_df","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/pipeline/data_struct.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/pipeline/data_struct.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a20d88089985381e"}},{"code_sha256_prefix":"f417d49ea2444696","entry":"convert_questions_to_dictlist","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/pipeline/data_struct.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/pipeline/data_struct.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f417d49ea2444696"}},{"code_sha256_prefix":"f31200d50375845a","entry":"transform_to_dict","repo":"maxlampe/mentat","repo_kind":"official","path":"mentat/eval_models/eval_openai.py","file_url":"https://github.com/maxlampe/mentat/blob/HEAD/mentat/eval_models/eval_openai.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f31200d50375845a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}