{"url":"/dataset/mediconfusion","name":"MediConfusion","full_name":null,"description_markdown":"MediConfusion is a challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical Multimodal Large Language Models (MLLMs) from a vision perspective. We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts.  <br />\r\nOur benchmark consists of 176 confusing pairs. A confusing pair is a set of two images that share the same question and corresponding answer options, but the correct answer is different for the images. <br />\r\nWe evaluate models based on their ability to answer <i>both</i> questions correctly within a confusing pair, which we call <b>set accuracy</b>. This metric indicates how well models can tell the two images apart, as a model that selects the same answer option for both images for all pairs will receive 0% set accuracy. We also report <b>confusion</b>, a metric that describes the proportion of confusing pairs where the model has chosen the same answer option for both images.","description_withheld":null,"homepage":"https://github.com/AIF4S/MediConfusion","introduced_date":"2024-09-23","introduced_date_note":null,"introduced_by":{"paper":"/paper/mediconfusion-can-you-trust-your-ai","title":"MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models","first_author":"Mohammad Shahab Sepehri","url":null},"license":{"name":"MIT","url":"https://opensource.org/license/MIT"},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"},{"name":"Biomedical","url":"/datasets/modality/biomedical"},{"name":"Medical","url":"/datasets/modality/medical"}],"tasks":[{"name":"Visual Question Answering (VQA)","url":"/task/visual-question-answering","datasets_with_task":"/datasets/task/visual-question-answering"},{"name":"Medical Visual Question Answering","url":"/task/medical-visual-question-answering","datasets_with_task":"/datasets/task/medical-visual-question-answering"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["MediConfusion"],"data_loaders":[{"repo":"https://github.com/AIF4S/MediConfusion","url":"https://github.com/AIF4S/MediConfusion","frameworks":["pytorch"]}],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}