{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-modal-learning-with-missing-modality-1","title":"Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling","arxiv_id":"2307.14126","date":"2023-07-26","proceeding":"CVPR 2023 1","authors":["Hu Wang","Yuanhong Chen","Congbo Ma","Jodie Avery","Louise Hull","Gustavo Carneiro"],"abstract":"The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour. The code repository address is https://github.com/billhhh/ShaSpec/.","url_abs":"https://arxiv.org/abs/2307.14126v2","url_pdf":"https://arxiv.org/pdf/2307.14126v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-modal-learning-with-missing-modality-1","repo_url":"https://github.com/billhhh/ShaSpec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"medical-image-segmentation","task_name":"Medical Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"domain-classification","task_name":"domain classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.14126","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.14126"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/billhhh/ShaSpec","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/cchen-cc/Robust-Mseg","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_draft_wrong":3,"ran_fixture":2,"unverified":8},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1},"found_in_text":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"f11a6e90cadb7f4a","entry":"ASPP","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f11a6e90cadb7f4a"}},{"code_sha256_prefix":"87a835e5bb115092","entry":"Activation_layer","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87a835e5bb115092"}},{"code_sha256_prefix":"aecac45f39b0b8d7","entry":"BasicBlock","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aecac45f39b0b8d7"}},{"code_sha256_prefix":"3c153f25c25aaeb4","entry":"Bottleneck","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c153f25c25aaeb4"}},{"code_sha256_prefix":"487b6da3a3ee0baf","entry":"CompositionalLayer","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"487b6da3a3ee0baf"}},{"code_sha256_prefix":"f87647971702596d","entry":"Conv3d_wd","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f87647971702596d"}},{"code_sha256_prefix":"6c8ddf5a387fa62b","entry":"Norm_layer","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c8ddf5a387fa62b"}},{"code_sha256_prefix":"7bbb9eb9394c1277","entry":"U_Res3D_dec","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bbb9eb9394c1277"}},{"code_sha256_prefix":"ab85f0066b239697","entry":"conv3x3x3","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab85f0066b239697"}},{"code_sha256_prefix":"43fa9247f4e0be35","entry":"decode_images","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43fa9247f4e0be35"}},{"code_sha256_prefix":"285c8c634525802b","entry":"downsample_basic_block","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"285c8c634525802b"}},{"code_sha256_prefix":"db52c4f2bee05325","entry":"kl_loss","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"losses.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db52c4f2bee05325"}},{"code_sha256_prefix":"976c8ef5d66a359c","entry":"plot_embedding_2D","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"976c8ef5d66a359c"}},{"code_sha256_prefix":"feb64926fe058ff2","entry":"DualNet_SS","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"feb64926fe058ff2"}},{"code_sha256_prefix":"c193cbe2e1e98ec6","entry":"ResNet","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c193cbe2e1e98ec6"}},{"code_sha256_prefix":"f871bd06635f4139","entry":"U_Res3D_enc","repo":"billhhh/ShaSpec","repo_kind":"official","path":"DualNet_SS.py","file_url":"https://github.com/billhhh/ShaSpec/blob/HEAD/DualNet_SS.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f871bd06635f4139"}},{"code_sha256_prefix":"ab241f8602321aab","entry":"decode_labels","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab241f8602321aab"}},{"code_sha256_prefix":"eebe5a9b8acdd970","entry":"dice_eval","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"stats_func.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/stats_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eebe5a9b8acdd970"}},{"code_sha256_prefix":"cc57c3ca7844ba89","entry":"generator","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"modalitydrop_generator.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/modalitydrop_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc57c3ca7844ba89"}},{"code_sha256_prefix":"69ee388442ee2275","entry":"load_data","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"data_loader.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69ee388442ee2275"}},{"code_sha256_prefix":"6620889b2afcdb57","entry":"task_loss","repo":"cchen-cc/Robust-Mseg","repo_kind":"found_in_text","path":"losses.py","file_url":"https://github.com/cchen-cc/Robust-Mseg/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6620889b2afcdb57"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}