{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contig-self-supervised-multimodal-contrastive","title":"ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics","arxiv_id":"2111.13424","date":"2021-11-26","proceeding":"CVPR 2022 1","authors":["Aiham Taleb","Matthias Kirchler","Remo Monti","Christoph Lippert"],"abstract":"High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalities vary across individuals. Our procedure outperforms state-of-the-art self-supervised methods on all evaluated downstream benchmark tasks. We also adapt gradient-based explainability algorithms to better understand the learned cross-modal associations between the images and genetic modalities. Finally, we perform genome-wide association studies on the features learned by our models, uncovering interesting relationships between images and genetic data.","url_abs":"https://arxiv.org/abs/2111.13424v1","url_pdf":"https://arxiv.org/pdf/2111.13424v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contig-self-supervised-multimodal-contrastive","repo_url":"https://github.com/HealthML/ContIG","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.13424","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.13424"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HealthML/ContIG","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4b60d529d194801c","entry":"acc_aptos","repo":"HealthML/ContIG","repo_kind":"official","path":"aptos_diabetic_retinopathy.py","file_url":"https://github.com/HealthML/ContIG/blob/HEAD/aptos_diabetic_retinopathy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b60d529d194801c"}},{"code_sha256_prefix":"4b3da4b334fb9c42","entry":"read_clumps","repo":"HealthML/ContIG","repo_kind":"official","path":"downstream_gwas.py","file_url":"https://github.com/HealthML/ContIG/blob/HEAD/downstream_gwas.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3da4b334fb9c42"}},{"code_sha256_prefix":"c5bf88719537d77d","entry":"score_kappa_aptos","repo":"HealthML/ContIG","repo_kind":"official","path":"aptos_diabetic_retinopathy.py","file_url":"https://github.com/HealthML/ContIG/blob/HEAD/aptos_diabetic_retinopathy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5bf88719537d77d"}},{"code_sha256_prefix":"62797f76def466a9","entry":"transform_multilabel_to_continuous","repo":"HealthML/ContIG","repo_kind":"official","path":"aptos_diabetic_retinopathy.py","file_url":"https://github.com/HealthML/ContIG/blob/HEAD/aptos_diabetic_retinopathy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62797f76def466a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}