{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmformer-multimodal-medical-transformer-for","title":"mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation","arxiv_id":"2206.02425","date":"2022-06-06","proceeding":null,"authors":["Yao Zhang","Nanjun He","Jiawei Yang","Yuexiang Li","Dong Wei","Yawen Huang","Yang Zhang","Zhiqiang He","Yefeng Zheng"],"abstract":"Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incomplete multimodal learning with three main components: the hybrid modality-specific encoders that bridge a convolutional encoder and an intra-modal Transformer for both local and global context modeling within each modality; an inter-modal Transformer to build and align the long-range correlations across modalities for modality-invariant features with global semantics corresponding to tumor region; a decoder that performs a progressive up-sampling and fusion with the modality-invariant features to generate robust segmentation. Besides, auxiliary regularizers are introduced in both encoder and decoder to further enhance the model's robustness to incomplete modalities. We conduct extensive experiments on the public BraTS $2018$ dataset for brain tumor segmentation. The results demonstrate that the proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities, especially by an average 19.07% improvement of Dice on tumor segmentation with only one available modality. The code is available at https://github.com/YaoZhang93/mmFormer.","url_abs":"https://arxiv.org/abs/2206.02425v2","url_pdf":"https://arxiv.org/pdf/2206.02425v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmformer-multimodal-medical-transformer-for","repo_url":"https://github.com/yaozhang93/mmformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"brain-tumor-segmentation","task_name":"Brain Tumor Segmentation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"tumor-segmentation","task_name":"Tumor Segmentation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"mmFormer","rank_in_archive_order":77,"of":121,"metrics":{"Mean IoU":"48.45%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.02425","atlas_url":"https://app.syntology.ai/?focus=2206.02425","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.02425"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaozhang93/mmformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":17,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fb26dcccfaf8fbc6","entry":"crop","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/preprocess.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb26dcccfaf8fbc6"}},{"code_sha256_prefix":"38db2c032df3a024","entry":"dice_loss","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/criterions.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38db2c032df3a024"}},{"code_sha256_prefix":"d51dae9ac1ba0359","entry":"get_params","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/lr_scheduler.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d51dae9ac1ba0359"}},{"code_sha256_prefix":"2675ee5f110411e2","entry":"get_temperature","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/lr_scheduler.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2675ee5f110411e2"}},{"code_sha256_prefix":"376f134d99d8ed75","entry":"normalization","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/layers.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/layers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"376f134d99d8ed75"}},{"code_sha256_prefix":"8ee9fb4494c15e91","entry":"normalize","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/preprocess.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ee9fb4494c15e91"}},{"code_sha256_prefix":"8d95807841fa86b8","entry":"parse","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/parser.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/parser.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d95807841fa86b8"}},{"code_sha256_prefix":"2c61eefd5b210b68","entry":"softmax_loss","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/criterions.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c61eefd5b210b68"}},{"code_sha256_prefix":"8383a6e515ab6db5","entry":"softmax_output_dice_class4","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/predict.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/predict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8383a6e515ab6db5"}},{"code_sha256_prefix":"4eb48bd860232e60","entry":"softmax_output_dice_class5","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/predict.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/predict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4eb48bd860232e60"}},{"code_sha256_prefix":"3a9b5bc021c9bc18","entry":"softmax_weighted_loss","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/criterions.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a9b5bc021c9bc18"}},{"code_sha256_prefix":"85878f80ff4c36c4","entry":"str2bool","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/str2bool.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/str2bool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85878f80ff4c36c4"}},{"code_sha256_prefix":"3153dadb37e140a0","entry":"sup_128","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/preprocess.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3153dadb37e140a0"}},{"code_sha256_prefix":"c3f09a933debc8e4","entry":"Js_div","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/lr_scheduler.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3f09a933debc8e4"}},{"code_sha256_prefix":"c765d38192a487f3","entry":"generate_snapshot","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/generate.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c765d38192a487f3"}},{"code_sha256_prefix":"c6d51b5a28f37da9","entry":"load","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/utils/parser.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/utils/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6d51b5a28f37da9"}},{"code_sha256_prefix":"ab7b9d7dda87dca9","entry":"test_softmax","repo":"yaozhang93/mmformer","repo_kind":"official","path":"mmformer/predict.py","file_url":"https://github.com/yaozhang93/mmformer/blob/HEAD/mmformer/predict.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab7b9d7dda87dca9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}