{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-learning-of-medical-visual","title":"Contrastive Learning of Medical Visual Representations from Paired Images and Text","arxiv_id":"2010.00747","date":"2020-10-02","proceeding":null,"authors":["Yuhao Zhang","Hang Jiang","Yasuhide Miura","Christopher D. Manning","Curtis P. Langlotz"],"abstract":"Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we find these methods help little on medical images because of their high inter-class similarity. We propose ConVIRT, an alternative unsupervised strategy to learn medical visual representations by exploiting naturally occurring paired descriptive text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test ConVIRT by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that it leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10\\% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.","url_abs":"https://arxiv.org/abs/2010.00747v2","url_pdf":"https://arxiv.org/pdf/2010.00747v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/yuhaozhang/convirt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/MicPie/clasp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/edreismd/convirt-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/edureisMD/ConVIRT-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/hemakaraveenhansika/convirt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/tjdevWorks/ConVIRT-Federated","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"contrastive-learning-of-medical-visual","repo_url":"https://github.com/jbdel/vilmedic","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"medical-image-classification","task_name":"Medical Image Classification"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2010.00747","atlas_url":"https://app.syntology.ai/?focus=2010.00747","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.00747"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tjdevWorks/ConVIRT-Federated","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/edureisMD/ConVIRT-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jbdel/vilmedic","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuhaozhang/convirt","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/edreismd/convirt-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hemakaraveenhansika/convirt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MicPie/clasp","reach":null}],"summary":{"ran":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"525d703da285cb16","entry":"get_params","repo":"tjdevWorks/ConVIRT-Federated","repo_kind":"listed","path":"src/flwr_client.py","file_url":"https://github.com/tjdevWorks/ConVIRT-Federated/blob/HEAD/src/flwr_client.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"525d703da285cb16"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}