{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-probabilistic-model-to-explain-self","title":"A Probabilistic Model Behind Self-Supervised Learning","arxiv_id":"2402.01399","date":"2024-02-02","proceeding":null,"authors":["Alice Bizeul","Bernhard Schölkopf","Carl Allen"],"abstract":"In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. A common task is to classify augmentations or different modalities of the data, which share semantic content (e.g. an object in an image) but differ in style (e.g. the object's location). Many approaches to self-supervised learning have been proposed, e.g. SimCLR, CLIP, and DINO, which have recently gained much attention for their representations achieving downstream performance comparable to supervised learning. However, a theoretical understanding of self-supervised methods eludes. Addressing this, we present a generative latent variable model for self-supervised learning and show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations, providing a unifying theoretical framework for these methods. The proposed model also justifies connections drawn to mutual information and the use of a ''projection head''. Learning representations by fitting the model generatively (termed SimVAE) improves performance over discriminative and other VAE-based methods on simple image benchmarks and significantly narrows the gap between generative and discriminative representation learning in more complex settings. Importantly, as our analysis predicts, SimVAE outperforms self-supervised learning where style information is required, taking an important step toward understanding self-supervised methods and achieving task-agnostic representations.","url_abs":"https://arxiv.org/abs/2402.01399v3","url_pdf":"https://arxiv.org/pdf/2402.01399v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-probabilistic-model-to-explain-self","repo_url":"https://github.com/alicebizeul/simvae","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"colorjitter","method_name":"ColorJitter"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dino","method_name":"DINO"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"nt-xent","method_name":"NT-Xent"},{"method_slug":"random-gaussian-blur","method_name":"Random Gaussian Blur"},{"method_slug":"random-resized-crop","method_name":"Random Resized Crop"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"simclr","method_name":"SimCLR"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01399","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01399"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alicebizeul/simvae","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"fa7588028e0141e1","entry":"cluster_acc","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa7588028e0141e1"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/resnet.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"af887bb1b2070bb9","entry":"conv1x1","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/decoders.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/decoders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af887bb1b2070bb9"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/resnet.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"086e14bf480ea57e","entry":"conv3x3","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/decoders.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/decoders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"086e14bf480ea57e"}},{"code_sha256_prefix":"55ad71f69f9ed31d","entry":"eval_classifier","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/models.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55ad71f69f9ed31d"}},{"code_sha256_prefix":"fc3f0eb9c010d7d4","entry":"eval_mlp","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/models.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc3f0eb9c010d7d4"}},{"code_sha256_prefix":"7e8b4582c6b69f22","entry":"plus_equals_nan","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e8b4582c6b69f22"}},{"code_sha256_prefix":"f02ceba98bd976b0","entry":"reparameterize","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/model/models.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/model/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f02ceba98bd976b0"}},{"code_sha256_prefix":"56ff7cdb48ee85b3","entry":"get_paths","repo":"alicebizeul/simvae","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/alicebizeul/simvae/blob/HEAD/src/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56ff7cdb48ee85b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}