{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/federated-modality-specific-encoders-and","title":"Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation","arxiv_id":"2403.11803","date":"2024-03-18","proceeding":null,"authors":["Qian Dai","Dong Wei","Hong Liu","Jinghan Sun","Liansheng Wang","Yefeng Zheng"],"abstract":"Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.","url_abs":"https://arxiv.org/abs/2403.11803v1","url_pdf":"https://arxiv.org/pdf/2403.11803v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"federated-modality-specific-encoders-and","repo_url":"https://github.com/qdaiing/fedmema","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"brain-tumor-segmentation","task_name":"Brain Tumor Segmentation"},{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"medical-image-analysis","task_name":"Medical Image Analysis"},{"task_slug":"tumor-segmentation","task_name":"Tumor Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.11803","atlas_url":"https://app.syntology.ai/?focus=2403.11803","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.11803"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/QDaiing/FedMEMA","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"cb69539b645f7473","entry":"Conv3D","repo":"QDaiing/FedMEMA","repo_kind":"official","path":"models/mmmodel.py","file_url":"https://github.com/QDaiing/FedMEMA/blob/HEAD/models/mmmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cb69539b645f7473"}},{"code_sha256_prefix":"0c3b78bfa896d587","entry":"Decoder","repo":"QDaiing/FedMEMA","repo_kind":"official","path":"models/mmmodel.py","file_url":"https://github.com/QDaiing/FedMEMA/blob/HEAD/models/mmmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0c3b78bfa896d587"}},{"code_sha256_prefix":"b8073bf178061209","entry":"Encoder","repo":"QDaiing/FedMEMA","repo_kind":"official","path":"models/mmmodel.py","file_url":"https://github.com/QDaiing/FedMEMA/blob/HEAD/models/mmmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b8073bf178061209"}},{"code_sha256_prefix":"7d0e939555cd1674","entry":"MMmodel","repo":"QDaiing/FedMEMA","repo_kind":"official","path":"models/mmmodel.py","file_url":"https://github.com/QDaiing/FedMEMA/blob/HEAD/models/mmmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7d0e939555cd1674"}},{"code_sha256_prefix":"ecdb1b00595256a9","entry":"ScaledDotProductAttention","repo":"QDaiing/FedMEMA","repo_kind":"official","path":"models/mmmodel.py","file_url":"https://github.com/QDaiing/FedMEMA/blob/HEAD/models/mmmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ecdb1b00595256a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}