{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-learning-inverts-the-data","title":"Contrastive Learning Inverts the Data Generating Process","arxiv_id":"2102.08850","date":"2021-02-17","proceeding":null,"authors":["Roland S. Zimmermann","Yash Sharma","Steffen Schneider","Matthias Bethge","Wieland Brendel"],"abstract":"Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.","url_abs":"https://arxiv.org/abs/2102.08850v4","url_pdf":"https://arxiv.org/pdf/2102.08850v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-learning-inverts-the-data","repo_url":"https://github.com/brendel-group/cl-ica","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"infonce","method_name":"InfoNCE"}],"datasets_introduced":[{"slug":"3dident","name":"3DIdent","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/disentanglement-on-3dident","task":"Disentanglement","dataset":"3DIdent","model":"InfoNCE (Normal, Box)","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"98.31"},"uses_additional_data":false},{"leaderboard":"/sota/disentanglement-on-kitti-masks","task":"Disentanglement","dataset":"KITTI-Masks","model":"InfoNCE (Laplace, Box)","rank_in_archive_order":1,"of":2,"metrics":{"MCC":"80.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.08850","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.08850"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brendel-group/cl-ica","reach":null}],"summary":{"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"efa370ec79f2dce2","entry":"uniform","repo":"brendel-group/cl-ica","repo_kind":"official","path":"main_kitti.py","file_url":"https://github.com/brendel-group/cl-ica/blob/HEAD/main_kitti.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efa370ec79f2dce2"}},{"code_sha256_prefix":"9958e08683dcf598","entry":"randint","repo":"brendel-group/cl-ica","repo_kind":"official","path":"main_kitti.py","file_url":"https://github.com/brendel-group/cl-ica/blob/HEAD/main_kitti.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9958e08683dcf598"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}