{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-optimal-transport-based-co","title":"Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction","arxiv_id":"2306.08330","date":"2023-06-14","proceeding":"ICCV 2023 1","authors":["Yingxue Xu","Hao Chen"],"abstract":"Survival prediction is a complicated ordinal regression task that aims to predict the ranking risk of death, which generally benefits from the integration of histology and genomic data. Despite the progress in joint learning from pathology and genomics, existing methods still suffer from challenging issues: 1) Due to the large size of pathological images, it is difficult to effectively represent the gigapixel whole slide images (WSIs). 2) Interactions within tumor microenvironment (TME) in histology are essential for survival analysis. Although current approaches attempt to model these interactions via co-attention between histology and genomic data, they focus on only dense local similarity across modalities, which fails to capture global consistency between potential structures, i.e. TME-related interactions of histology and co-expression of genomic data. To address these challenges, we propose a Multimodal Optimal Transport-based Co-Attention Transformer framework with global structure consistency, in which optimal transport (OT) is applied to match patches of a WSI and genes embeddings for selecting informative patches to represent the gigapixel WSI. More importantly, OT-based co-attention provides a global awareness to effectively capture structural interactions within TME for survival prediction. To overcome high computational complexity of OT, we propose a robust and efficient implementation over micro-batch of WSI patches by approximating the original OT with unbalanced mini-batch OT. Extensive experiments show the superiority of our method on five benchmark datasets compared to the state-of-the-art methods. The code is released.","url_abs":"https://arxiv.org/abs/2306.08330v2","url_pdf":"https://arxiv.org/pdf/2306.08330v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-optimal-transport-based-co","repo_url":"https://github.com/innse/motcat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"multimodal-optimal-transport-based-co","repo_url":"https://github.com/JJ-ZHOU-Code/DAMM-Diffusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"multimodal-optimal-transport-based-co","repo_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"survival-analysis","task_name":"Survival Analysis"},{"task_slug":"survival-prediction","task_name":"Survival Prediction"},{"task_slug":"whole-slide-images","task_name":"whole slide images"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.08330","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.08330"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JJ-ZHOU-Code/DAMM-Diffusion","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/innse/motcat","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel","reach":null}],"summary":{"ran":7,"ran_draft_wrong":2,"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1},"listed":{"samples":16,"ran":10,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"dbd80cddba6f67f2","entry":"Attn_Net_Gated","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dbd80cddba6f67f2"}},{"code_sha256_prefix":"314f5264c2883265","entry":"BilinearFusion","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"314f5264c2883265"}},{"code_sha256_prefix":"b8a656d552c1f7f0","entry":"NLLSurvLoss","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8a656d552c1f7f0"}},{"code_sha256_prefix":"d4e04018342ece86","entry":"NystromAttention","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4e04018342ece86"}},{"code_sha256_prefix":"7cc8eaf772224044","entry":"PPEG","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cc8eaf772224044"}},{"code_sha256_prefix":"7c89241e7e5aff7f","entry":"SCAF","repo":"JJ-ZHOU-Code/DAMM-Diffusion","repo_kind":"listed","path":"diffusion/models/utils/my_attention.py","file_url":"https://github.com/JJ-ZHOU-Code/DAMM-Diffusion/blob/HEAD/diffusion/models/utils/my_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c89241e7e5aff7f"}},{"code_sha256_prefix":"f2b09ae164d718b5","entry":"SE_Layer","repo":"JJ-ZHOU-Code/DAMM-Diffusion","repo_kind":"listed","path":"diffusion/models/utils/my_attention.py","file_url":"https://github.com/JJ-ZHOU-Code/DAMM-Diffusion/blob/HEAD/diffusion/models/utils/my_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2b09ae164d718b5"}},{"code_sha256_prefix":"98bab5b56a287e4d","entry":"calc_W_loss","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98bab5b56a287e4d"}},{"code_sha256_prefix":"d9f70b6ed26bc6dc","entry":"calc_kl_divergence","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9f70b6ed26bc6dc"}},{"code_sha256_prefix":"62ede8fe77ab4913","entry":"nll_loss","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62ede8fe77ab4913"}},{"code_sha256_prefix":"82db0b1414d840c6","entry":"MOTCAT_Surv","repo":"innse/motcat","repo_kind":"official","path":"models/model_motcat.py","file_url":"https://github.com/innse/motcat/blob/HEAD/models/model_motcat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82db0b1414d840c6"}},{"code_sha256_prefix":"b3b023b4c6805594","entry":"MultiheadAttention","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3b023b4c6805594"}},{"code_sha256_prefix":"eee6a34fb548d0d6","entry":"OT_Attn_assem","repo":"innse/motcat","repo_kind":"official","path":"models/model_motcat.py","file_url":"https://github.com/innse/motcat/blob/HEAD/models/model_motcat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eee6a34fb548d0d6"}},{"code_sha256_prefix":"df3c40ea28eee3ff","entry":"Robust_MCAT","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df3c40ea28eee3ff"}},{"code_sha256_prefix":"22eac946d1b47bb1","entry":"TransLayer","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22eac946d1b47bb1"}},{"code_sha256_prefix":"c41f70496ff795c3","entry":"VIBTrans","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c41f70496ff795c3"}},{"code_sha256_prefix":"763d48b1144254cd","entry":"moore_penrose_iter_pinv","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"763d48b1144254cd"}},{"code_sha256_prefix":"4043b3b3f31ca073","entry":"multi_head_attention_forward","repo":"JJ-ZHOU-Code/RobustMultiModel","repo_kind":"listed","path":"models/model_coattn.py","file_url":"https://github.com/JJ-ZHOU-Code/RobustMultiModel/blob/HEAD/models/model_coattn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4043b3b3f31ca073"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}