{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/medical-diagnosis-with-large-scale-multimodal","title":"Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis","arxiv_id":"2212.09162","date":"2022-12-18","proceeding":null,"authors":["Firas Khader","Gustav Mueller-Franzes","Tianci Wang","Tianyu Han","Soroosh Tayebi Arasteh","Christoph Haarburger","Johannes Stegmaier","Keno Bressem","Christiane Kuhl","Sven Nebelung","Jakob Nikolas Kather","Daniel Truhn"],"abstract":"Multimodal deep learning has been used to predict clinical endpoints and diagnoses from clinical routine data. However, these models suffer from scaling issues: they have to learn pairwise interactions between each piece of information in each data type, thereby escalating model complexity beyond manageable scales. This has so far precluded a widespread use of multimodal deep learning. Here, we present a new technical approach of \"learnable synergies\", in which the model only selects relevant interactions between data modalities and keeps an \"internal memory\" of relevant data. Our approach is easily scalable and naturally adapts to multimodal data inputs from clinical routine. We demonstrate this approach on three large multimodal datasets from radiology and ophthalmology and show that it outperforms state-of-the-art models in clinically relevant diagnosis tasks. Our new approach is transferable and will allow the application of multimodal deep learning to a broad set of clinically relevant problems.","url_abs":"https://arxiv.org/abs/2212.09162v2","url_pdf":"https://arxiv.org/pdf/2212.09162v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"medical-diagnosis-with-large-scale-multimodal","repo_url":"https://github.com/firasgit/lsmt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"medical-diagnosis","task_name":"Medical Diagnosis"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}