{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/passion-towards-effective-incomplete-multi","title":"PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates","arxiv_id":"2407.14796","date":"2024-07-20","proceeding":null,"authors":["Junjie Shi","Caozhi Shang","Zhaobin Sun","Li Yu","Xin Yang","Zengqiang Yan"],"abstract":"Incomplete multi-modal image segmentation is a fundamental task in medical imaging to refine deployment efficiency when only partial modalities are available. However, the common practice that complete-modality data is visible during model training is far from realistic, as modalities can have imbalanced missing rates in clinical scenarios. In this paper, we, for the first time, formulate such a challenging setting and propose Preference-Aware Self-diStillatION (PASSION) for incomplete multi-modal medical image segmentation under imbalanced missing rates. Specifically, we first construct pixel-wise and semantic-wise self-distillation to balance the optimization objective of each modality. Then, we define relative preference to evaluate the dominance of each modality during training, based on which to design task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate the superiority of PASSION against existing approaches for modality balancing. More importantly, PASSION is validated to work as a plug-and-play module for consistent performance improvement across different backbones. Code is available at https://github.com/Jun-Jie-Shi/PASSION.","url_abs":"https://arxiv.org/abs/2407.14796v1","url_pdf":"https://arxiv.org/pdf/2407.14796v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"passion-towards-effective-incomplete-multi","repo_url":"https://github.com/jun-jie-shi/passion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"medical-image-segmentation","task_name":"Medical Image Segmentation"},{"task_slug":"multi-modal-image-segmentation","task_name":"Multi-modal image segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.14796","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.14796"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jun-jie-shi/passion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":15,"unverified":1},"by_repo_kind":{"official":{"samples":16,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c2097366a0c9a24","entry":"Js_div","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/lr_scheduler.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/lr_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c2097366a0c9a24"}},{"code_sha256_prefix":"fb26dcccfaf8fbc6","entry":"crop","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/preprocessing/preprocess_brats.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/preprocessing/preprocess_brats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb26dcccfaf8fbc6"}},{"code_sha256_prefix":"672eefec7e9131bc","entry":"dice_loss","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/criterions.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/criterions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"672eefec7e9131bc"}},{"code_sha256_prefix":"873433db15bc1e4d","entry":"dice_loss_bs","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/criterions.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/criterions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"873433db15bc1e4d"}},{"code_sha256_prefix":"d51dae9ac1ba0359","entry":"get_params","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/lr_scheduler.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/lr_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d51dae9ac1ba0359"}},{"code_sha256_prefix":"2675ee5f110411e2","entry":"get_temperature","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/lr_scheduler.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/lr_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2675ee5f110411e2"}},{"code_sha256_prefix":"7f3257ab27f6ac4d","entry":"mask_gen_cross4","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/models/mask.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/models/mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f3257ab27f6ac4d"}},{"code_sha256_prefix":"35e8e3aef9fcc772","entry":"mask_gen_fusion","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/models/mask.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/models/mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35e8e3aef9fcc772"}},{"code_sha256_prefix":"c7dfb8d3e7844879","entry":"nchwd2nlc2nchwd","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/models/blocks.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/models/blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7dfb8d3e7844879"}},{"code_sha256_prefix":"268e21ea88a8ccb1","entry":"normalization","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/models/blocks.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/models/blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"268e21ea88a8ccb1"}},{"code_sha256_prefix":"8ee9fb4494c15e91","entry":"normalize","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/preprocessing/preprocess_brats.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/preprocessing/preprocess_brats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ee9fb4494c15e91"}},{"code_sha256_prefix":"8a289ba730141ed4","entry":"parse","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/parser.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a289ba730141ed4"}},{"code_sha256_prefix":"b5ee99865309afa1","entry":"softmax_weighted_loss","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/criterions.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/criterions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5ee99865309afa1"}},{"code_sha256_prefix":"4663b0d3de392cbf","entry":"split_data","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/preprocessing/data_split.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/preprocessing/data_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4663b0d3de392cbf"}},{"code_sha256_prefix":"3153dadb37e140a0","entry":"sup_128","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/preprocessing/preprocess_brats.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/preprocessing/preprocess_brats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3153dadb37e140a0"}},{"code_sha256_prefix":"b40f27cd88000a35","entry":"load","repo":"jun-jie-shi/passion","repo_kind":"official","path":"code/utils/parser.py","file_url":"https://github.com/jun-jie-shi/passion/blob/HEAD/code/utils/parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b40f27cd88000a35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}