{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-generalist-foundation-model-for","title":"Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data","arxiv_id":"2308.02463","date":"2023-08-04","proceeding":null,"authors":["Chaoyi Wu","Xiaoman Zhang","Ya zhang","Yanfeng Wang","Weidi Xie"],"abstract":"In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.","url_abs":"https://arxiv.org/abs/2308.02463v5","url_pdf":"https://arxiv.org/pdf/2308.02463v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-generalist-foundation-model-for","repo_url":"https://github.com/chaoyi-wu/radfm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.02463","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.02463"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chaoyi-wu/radfm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":2,"ran_violates":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d75570b4e91d0b4","entry":"FeedForward","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/helpers.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d75570b4e91d0b4"}},{"code_sha256_prefix":"c8f7b3c548c9d56a","entry":"build_position_encoding","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/position_encoding.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/position_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8f7b3c548c9d56a"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/helpers.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"fe68fff801021b23","entry":"getattr_recursive","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/utils.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe68fff801021b23"}},{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/vit_3d.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/vit_3d.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}},{"code_sha256_prefix":"5eab665292b38cf2","entry":"vision_load_pretrain","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/utils.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5eab665292b38cf2"}},{"code_sha256_prefix":"fb9045fc9e031a91","entry":"get_visual_encoder","repo":"chaoyi-wu/radfm","repo_kind":"official","path":"Quick_demo/Model/RadFM/utils.py","file_url":"https://github.com/chaoyi-wu/radfm/blob/HEAD/Quick_demo/Model/RadFM/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb9045fc9e031a91"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}