{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-contrastive-learning-for-spatial","title":"Multimodal contrastive learning for spatial gene expression prediction using histology images","arxiv_id":"2407.08216","date":"2024-07-11","proceeding":null,"authors":["Wenwen Min","Zhiceng Shi","Jun Zhang","Jun Wan","Changmiao Wang"],"abstract":"In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images (WSIs) stained with Hematoxylin and Eosin (H\\&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose \\textbf{mclSTExp}, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a \"word\", integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. Our extensive evaluation of \\textbf{mclSTExp} on two breast cancer datasets and a skin squamous cell carcinoma dataset demonstrates its superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.","url_abs":"https://arxiv.org/abs/2407.08216v1","url_pdf":"https://arxiv.org/pdf/2407.08216v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-contrastive-learning-for-spatial","repo_url":"https://github.com/shizhiceng/mclstexp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"whole-slide-images","task_name":"whole slide images"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.08216","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08216"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shizhiceng/mclstexp","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":1,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"62871b7ee585cf51","entry":"NB_loss","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"baselines/His2ST/NB_module.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/baselines/His2ST/NB_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62871b7ee585cf51"}},{"code_sha256_prefix":"01409d6c760c9feb","entry":"ZINB_loss","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"baselines/His2ST/NB_module.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/baselines/His2ST/NB_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01409d6c760c9feb"}},{"code_sha256_prefix":"63e6e57b587c3aa8","entry":"cross_entropy","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"baselines/Bleep/models.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/baselines/Bleep/models.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63e6e57b587c3aa8"}},{"code_sha256_prefix":"576657ded1637110","entry":"find_matches","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"evel_cscc.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/evel_cscc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"576657ded1637110"}},{"code_sha256_prefix":"fa4dd37d2a9647a4","entry":"find_matches","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"evel_her2st.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/evel_her2st.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa4dd37d2a9647a4"}},{"code_sha256_prefix":"7cdb01bb7f8f64bf","entry":"get_R","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"evel_cscc.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/evel_cscc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cdb01bb7f8f64bf"}},{"code_sha256_prefix":"7337f1f5ff01dcd0","entry":"get_lr","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7337f1f5ff01dcd0"}},{"code_sha256_prefix":"48fcf5eb5babda05","entry":"intersect_section_genes","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"hvg_cscc.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/hvg_cscc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48fcf5eb5babda05"}},{"code_sha256_prefix":"7f0d0b449663bc7c","entry":"her2_pool_gene_list","repo":"shizhiceng/mclstexp","repo_kind":"official","path":"hvg_cscc.py","file_url":"https://github.com/shizhiceng/mclstexp/blob/HEAD/hvg_cscc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f0d0b449663bc7c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}