{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/med3dvlm-an-efficient-vision-language-model","title":"Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis","arxiv_id":"2503.20047","date":"2025-03-25","proceeding":null,"authors":["Yu Xin","Gorkem Can Ates","Kuang Gong","Wei Shao"],"abstract":"Vision-language models (VLMs) have shown promise in 2D medical image analysis, but extending them to 3D remains challenging due to the high computational demands of volumetric data and the difficulty of aligning 3D spatial features with clinical text. We present Med3DVLM, a 3D VLM designed to address these challenges through three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture fine-grained spatial features at scale; (2) SigLIP, a contrastive learning strategy with pairwise sigmoid loss that improves image-text alignment without relying on large negative batches; and (3) a dual-stream MLP-Mixer projector that fuses low- and high-level image features with text embeddings for richer multi-modal representations. We evaluate our model on the M3D dataset, which includes radiology reports and VQA data for 120,084 3D medical images. Results show that Med3DVLM achieves superior performance across multiple benchmarks. For image-text retrieval, it reaches 61.00% R@1 on 2,000 samples, significantly outperforming the current state-of-the-art M3D model (19.10%). For report generation, it achieves a METEOR score of 36.42% (vs. 14.38%). In open-ended visual question answering (VQA), it scores 36.76% METEOR (vs. 33.58%), and in closed-ended VQA, it achieves 79.95% accuracy (vs. 75.78%). These results highlight Med3DVLM's ability to bridge the gap between 3D imaging and language, enabling scalable, multi-task reasoning across clinical applications. Our code is publicly available at https://github.com/mirthAI/Med3DVLM.","url_abs":"https://arxiv.org/abs/2503.20047v1","url_pdf":"https://arxiv.org/pdf/2503.20047v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"med3dvlm-an-efficient-vision-language-model","repo_url":"https://github.com/mirthai/med3dvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"medical-image-analysis","task_name":"Medical Image Analysis"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"mlp-mixer","method_name":"MLP-Mixer"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.20047","atlas_url":"https://app.syntology.ai/?focus=2503.20047","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.20047"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mirthai/med3dvlm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4a4895248ff31778","entry":"calculate_precision","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_clip.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_clip.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a4895248ff31778"}},{"code_sha256_prefix":"347df39071b0661a","entry":"calculate_recall","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_clip.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_clip.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"347df39071b0661a"}},{"code_sha256_prefix":"220b7697c5a0d40e","entry":"compute_metrics","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/train/train_clip.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/train/train_clip.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"220b7697c5a0d40e"}},{"code_sha256_prefix":"05b8d6cd94313722","entry":"compute_metrics","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/train/train_vlm.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/train/train_vlm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b8d6cd94313722"}},{"code_sha256_prefix":"8b51adb8a6c014e5","entry":"postprocess_text","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_caption.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_caption.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b51adb8a6c014e5"}},{"code_sha256_prefix":"0b66a8919774c46f","entry":"preprocess_logits_for_metrics","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/train/train_vlm.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/train/train_vlm.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b66a8919774c46f"}},{"code_sha256_prefix":"796e2d1861bc5152","entry":"preprocess_logits_for_metrics","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/train/train_clip.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/train/train_clip.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"796e2d1861bc5152"}},{"code_sha256_prefix":"f78bb358bb60eef1","entry":"gather_features","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/model/CLIP.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/model/CLIP.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f78bb358bb60eef1"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/train/train_vlm.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/train/train_vlm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"9ab24b5ed20f3284","entry":"parse_args","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_caption.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_caption.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ab24b5ed20f3284"}},{"code_sha256_prefix":"c954ff9bd4957277","entry":"parse_args","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_clip.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c954ff9bd4957277"}},{"code_sha256_prefix":"ee98572d724bc5c7","entry":"parse_args","repo":"mirthai/med3dvlm","repo_kind":"official","path":"src/eval/eval_vqa.py","file_url":"https://github.com/mirthai/med3dvlm/blob/HEAD/src/eval/eval_vqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee98572d724bc5c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}