{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmfusion-multi-modality-diffusion-model-for","title":"MMFusion: Multi-modality Diffusion Model for Lymph Node Metastasis Diagnosis in Esophageal Cancer","arxiv_id":"2405.09539","date":"2024-05-15","proceeding":null,"authors":["Chengyu Wu","Chengkai Wang","Yaqi Wang","Huiyu Zhou","Yatao Zhang","Qifeng Wang","Shuai Wang"],"abstract":"Esophageal cancer is one of the most common types of cancer worldwide and ranks sixth in cancer-related mortality. Accurate computer-assisted diagnosis of cancer progression can help physicians effectively customize personalized treatment plans. Currently, CT-based cancer diagnosis methods have received much attention for their comprehensive ability to examine patients' conditions. However, multi-modal based methods may likely introduce information redundancy, leading to underperformance. In addition, efficient and effective interactions between multi-modal representations need to be further explored, lacking insightful exploration of prognostic correlation in multi-modality features. In this work, we introduce a multi-modal heterogeneous graph-based conditional feature-guided diffusion model for lymph node metastasis diagnosis based on CT images as well as clinical measurements and radiomics data. To explore the intricate relationships between multi-modal features, we construct a heterogeneous graph. Following this, a conditional feature-guided diffusion approach is applied to eliminate information redundancy. Moreover, we propose a masked relational representation learning strategy, aiming to uncover the latent prognostic correlations and priorities of primary tumor and lymph node image representations. Various experimental results validate the effectiveness of our proposed method. The code is available at https://github.com/wuchengyu123/MMFusion.","url_abs":"https://arxiv.org/abs/2405.09539v2","url_pdf":"https://arxiv.org/pdf/2405.09539v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmfusion-multi-modality-diffusion-model-for","repo_url":"https://github.com/wuchengyu123/mmfusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.09539","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.09539"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wuchengyu123/mmfusion","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"bb3f06949955297c","entry":"attention","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/cross_relation_atten.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/cross_relation_atten.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb3f06949955297c"}},{"code_sha256_prefix":"7dbec2e994277534","entry":"compute_metrics","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"utils.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7dbec2e994277534"}},{"code_sha256_prefix":"132caaf49db1e0aa","entry":"conv3x3x3","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"132caaf49db1e0aa"}},{"code_sha256_prefix":"03e03bbdf5410afc","entry":"downsample_basic_block","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03e03bbdf5410afc"}},{"code_sha256_prefix":"95f06c920b5d6995","entry":"make_beta_schedule","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/diffusion_utils.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/diffusion_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95f06c920b5d6995"}},{"code_sha256_prefix":"4a7a1e2ec9717491","entry":"masked_attention","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/cross_relation_atten.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/cross_relation_atten.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a7a1e2ec9717491"}},{"code_sha256_prefix":"3521e17a155ed25a","entry":"q_sample","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/diffusion_utils.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/diffusion_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3521e17a155ed25a"}},{"code_sha256_prefix":"adf15d4407489512","entry":"randbool","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/cross_relation_atten.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/cross_relation_atten.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adf15d4407489512"}},{"code_sha256_prefix":"a79e3d02137db02f","entry":"create_fully_connected_graph","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/BuildModel.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/BuildModel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a79e3d02137db02f"}},{"code_sha256_prefix":"30e33c6c36337446","entry":"extract","repo":"wuchengyu123/mmfusion","repo_kind":"official","path":"model/diffusion_utils.py","file_url":"https://github.com/wuchengyu123/mmfusion/blob/HEAD/model/diffusion_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30e33c6c36337446"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}