{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-vision-language-models-ready-for-clinical","title":"Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering","arxiv_id":"2505.18915","date":"2025-05-25","proceeding":null,"authors":["Yixiong Chen","Wenjie Xiao","Pedro R. A. S. Bassi","Xinze Zhou","Sezgin Er","Ibrahim Ethem Hamamci","Zongwei Zhou","Alan Yuille"],"abstract":"Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.","url_abs":"https://arxiv.org/abs/2505.18915v1","url_pdf":"https://arxiv.org/pdf/2505.18915v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-vision-language-models-ready-for-clinical","repo_url":"https://github.com/schuture/deeptumorvqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"anatomy","task_name":"Anatomy"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.18915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.18915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Schuture/DeepTumorVQA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/schuture/deeptumorvqa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6d7b0289b78ca830","entry":"extract_letter","repo":"Schuture/DeepTumorVQA","repo_kind":"official","path":"src/deeptumorvqa/eval/metrics.py","file_url":"https://github.com/Schuture/DeepTumorVQA/blob/HEAD/src/deeptumorvqa/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d7b0289b78ca830"}},{"code_sha256_prefix":"b35d31006104e7a1","entry":"extract_number","repo":"Schuture/DeepTumorVQA","repo_kind":"official","path":"src/deeptumorvqa/eval/metrics.py","file_url":"https://github.com/Schuture/DeepTumorVQA/blob/HEAD/src/deeptumorvqa/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b35d31006104e7a1"}},{"code_sha256_prefix":"6c1fa66eefd73401","entry":"format_ranking","repo":"Schuture/DeepTumorVQA","repo_kind":"official","path":"src/deeptumorvqa/eval/leaderboard.py","file_url":"https://github.com/Schuture/DeepTumorVQA/blob/HEAD/src/deeptumorvqa/eval/leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c1fa66eefd73401"}},{"code_sha256_prefix":"b03a71d33c9198a3","entry":"insert_user_model","repo":"Schuture/DeepTumorVQA","repo_kind":"official","path":"src/deeptumorvqa/eval/leaderboard.py","file_url":"https://github.com/Schuture/DeepTumorVQA/blob/HEAD/src/deeptumorvqa/eval/leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b03a71d33c9198a3"}},{"code_sha256_prefix":"089e2a1ef1bdb07a","entry":"strip_think","repo":"Schuture/DeepTumorVQA","repo_kind":"official","path":"src/deeptumorvqa/eval/metrics.py","file_url":"https://github.com/Schuture/DeepTumorVQA/blob/HEAD/src/deeptumorvqa/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"089e2a1ef1bdb07a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}