{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-learning-with-data","title":"Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style","arxiv_id":"2106.04619","date":"2021-06-08","proceeding":"NeurIPS 2021 12","authors":["Julius von Kügelgen","Yash Sharma","Luigi Gresele","Wieland Brendel","Bernhard Schölkopf","Michel Besserve","Francesco Locatello"],"abstract":"Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.","url_abs":"https://arxiv.org/abs/2106.04619v4","url_pdf":"https://arxiv.org/pdf/2106.04619v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-with-data","repo_url":"https://github.com/ysharma1126/ssl_identifiability","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[{"slug":"causal3dident","name":"Causal3DIdent","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-causal3dident","task":"Image Classification","dataset":"Causal3DIdent","model":"SimCLR","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy ":"1.00"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-causal3dident","task":"Image Classification","dataset":"Causal3DIdent","model":"Barlow Twins","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy ":"0.99"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04619","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04619"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ysharma1126/ssl_identifiability","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b7c99fad0c328715","entry":"get_exp_name","repo":"ysharma1126/ssl_identifiability","repo_kind":"official","path":"main_mlp.py","file_url":"https://github.com/ysharma1126/ssl_identifiability/blob/HEAD/main_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7c99fad0c328715"}},{"code_sha256_prefix":"def79ba06f792d6f","entry":"get_exp_name","repo":"ysharma1126/ssl_identifiability","repo_kind":"official","path":"main_3dident.py","file_url":"https://github.com/ysharma1126/ssl_identifiability/blob/HEAD/main_3dident.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"def79ba06f792d6f"}},{"code_sha256_prefix":"38f4245139bfc92d","entry":"valid_str","repo":"ysharma1126/ssl_identifiability","repo_kind":"official","path":"main_mlp.py","file_url":"https://github.com/ysharma1126/ssl_identifiability/blob/HEAD/main_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38f4245139bfc92d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}