{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2408-02900","title":"MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine","arxiv_id":"2408.02900","date":"2024-08-06","proceeding":null,"authors":["Yunfei Xie","Ce Zhou","Lang Gao","Juncheng Wu","Xianhang Li","Hong-Yu Zhou","Sheng Liu","Lei Xing","James Zou","Cihang Xie","Yuyin Zhou"],"abstract":"This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available.","url_abs":"https://arxiv.org/abs/2408.02900v2","url_pdf":"https://arxiv.org/pdf/2408.02900v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2408-02900","repo_url":"https://github.com/UCSC-VLAA/MedTrinity-25M","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"medical-visual-question-answering","task_name":"Medical Visual Question Answering"},{"task_slug":"organ-detection","task_name":"Organ Detection"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[{"slug":"medtrinity-25m","name":"MedTrinity-25M","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.02900","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.02900"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UCSC-VLAA/MedTrinity-25M","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"bc2bc6de32629ca8","entry":"get_all_images","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"data_process/cellpose_infer_batch.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/data_process/cellpose_infer_batch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc2bc6de32629ca8"}},{"code_sha256_prefix":"dd3b9c964c5d49ff","entry":"get_bounding_boxes_and_save_wmask","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"data_process/cellpose_infer_batch.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/data_process/cellpose_infer_batch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd3b9c964c5d49ff"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"0322f102a810d3e0","entry":"norm01","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"data_process/cellpose_infer_batch.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/data_process/cellpose_infer_batch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0322f102a810d3e0"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"UCSC-VLAA/MedTrinity-25M","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/UCSC-VLAA/MedTrinity-25M/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}