{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transcriptomics-guided-slide-representation","title":"Transcriptomics-guided Slide Representation Learning in Computational Pathology","arxiv_id":"2405.11618","date":"2024-05-19","proceeding":"CVPR 2024 1","authors":["Guillaume Jaume","Lukas Oldenburg","Anurag Vaidya","Richard J. Chen","Drew F. K. Williamson","Thomas Peeters","Andrew H. Song","Faisal Mahmood"],"abstract":"Self-supervised learning (SSL) has been successful in building patch embeddings of small histology images (e.g., 224x224 pixels), but scaling these models to learn slide embeddings from the entirety of giga-pixel whole-slide images (WSIs) remains challenging. Here, we leverage complementary information from gene expression profiles to guide slide representation learning using multimodal pre-training. Expression profiles constitute highly detailed molecular descriptions of a tissue that we hypothesize offer a strong task-agnostic training signal for learning slide embeddings. Our slide and expression (S+E) pre-training strategy, called Tangle, employs modality-specific encoders, the outputs of which are aligned via contrastive learning. Tangle was pre-trained on samples from three different organs: liver (n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different species (Homo sapiens and Rattus norvegicus). Across three independent test datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs, Tangle shows significantly better few-shot performance compared to supervised and SSL baselines. When assessed using prototype-based classification and slide retrieval, Tangle also shows a substantial performance improvement over all baselines. Code available at https://github.com/mahmoodlab/TANGLE.","url_abs":"https://arxiv.org/abs/2405.11618v1","url_pdf":"https://arxiv.org/pdf/2405.11618v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transcriptomics-guided-slide-representation","repo_url":"https://github.com/mahmoodlab/tangle","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"whole-slide-images","task_name":"whole slide images"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mahmoodlab/tangle","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mahmoodlab/TANGLE","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":14,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"4b8cd06d1b3f2f1d","entry":"ABMILEmbedder","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4b8cd06d1b3f2f1d"}},{"code_sha256_prefix":"b9ef1c4dffe4e7ba","entry":"ABMILEmbedder_MH","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b9ef1c4dffe4e7ba"}},{"code_sha256_prefix":"632935273ac99ca3","entry":"BatchedABMIL","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"632935273ac99ca3"}},{"code_sha256_prefix":"caf1ffd214ce1f37","entry":"MLP","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"caf1ffd214ce1f37"}},{"code_sha256_prefix":"9d6c31fffb67a0e0","entry":"ProjHead","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d6c31fffb67a0e0"}},{"code_sha256_prefix":"7cca15b9821296c4","entry":"apply_random_mask","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/loss/tangle_loss.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/loss/tangle_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7cca15b9821296c4"}},{"code_sha256_prefix":"dc22df19cd5e4ef3","entry":"calculate_metrics","repo":"mahmoodlab/tangle","repo_kind":"official","path":"run_linear_probing.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/run_linear_probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dc22df19cd5e4ef3"}},{"code_sha256_prefix":"a014e3f1a7d5aba2","entry":"collate_slide","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/utils/learning.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/utils/learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a014e3f1a7d5aba2"}},{"code_sha256_prefix":"2e5d9d19ffa6e348","entry":"collate_tangle","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/utils/learning.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/utils/learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2e5d9d19ffa6e348"}},{"code_sha256_prefix":"e0c8ee1b7aef6f88","entry":"load_and_split","repo":"mahmoodlab/tangle","repo_kind":"official","path":"run_linear_probing.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/run_linear_probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e0c8ee1b7aef6f88"}},{"code_sha256_prefix":"b5007322c2f2ec6e","entry":"load_h5","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/dataset/dataset.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/dataset/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5007322c2f2ec6e"}},{"code_sha256_prefix":"369ca48ec835bd84","entry":"load_pkl","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/utils/learning.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/utils/learning.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"369ca48ec835bd84"}},{"code_sha256_prefix":"e37cfb34afd06711","entry":"read_config","repo":"mahmoodlab/tangle","repo_kind":"official","path":"extract_slide_embeddings_from_checkpoint.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/extract_slide_embeddings_from_checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e37cfb34afd06711"}},{"code_sha256_prefix":"6fbfbf2f5b75e23b","entry":"restore_model","repo":"mahmoodlab/tangle","repo_kind":"official","path":"extract_slide_embeddings_from_checkpoint.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/extract_slide_embeddings_from_checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6fbfbf2f5b75e23b"}},{"code_sha256_prefix":"dbc2525b2c770e82","entry":"set_args","repo":"mahmoodlab/tangle","repo_kind":"official","path":"extract_slide_embeddings_from_checkpoint.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/extract_slide_embeddings_from_checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dbc2525b2c770e82"}},{"code_sha256_prefix":"bae8b1daa1bb6af4","entry":"MMSSL","repo":"mahmoodlab/TANGLE","repo_kind":"official","path":"core/models/mmssl.py","file_url":"https://github.com/mahmoodlab/TANGLE/blob/HEAD/core/models/mmssl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bae8b1daa1bb6af4"}},{"code_sha256_prefix":"0215414a4311e93e","entry":"init_intra_wsi_loss_function","repo":"mahmoodlab/tangle","repo_kind":"official","path":"core/loss/tangle_loss.py","file_url":"https://github.com/mahmoodlab/tangle/blob/HEAD/core/loss/tangle_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0215414a4311e93e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}