{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-self-supervised-learning","title":"Understanding self-supervised Learning Dynamics without Contrastive Pairs","arxiv_id":"2102.06810","date":"2021-02-12","proceeding":null,"authors":["Yuandong Tian","Xinlei Chen","Surya Ganguli"],"abstract":"While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \\emph{non-contrastive} SSL (e.g., BYOL and SimSiam) show remarkable performance {\\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that \\emph{directly} sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by $2.5\\%$ in 300-epoch training (and $5\\%$ in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.","url_abs":"https://arxiv.org/abs/2102.06810v4","url_pdf":"https://arxiv.org/pdf/2102.06810v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-self-supervised-learning","repo_url":"https://github.com/facebookresearch/luckmatters","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"understanding-self-supervised-learning","repo_url":"https://github.com/facebookresearch/luckmatters/tree/master/ssl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"understanding-self-supervised-learning","repo_url":"https://github.com/fmi-basel/implicit-var-reg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"understanding-self-supervised-learning","repo_url":"https://github.com/lucidrains/byol-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"understanding-self-supervised-learning","repo_url":"https://github.com/miszkur/SelfSupervisedLearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"byol","method_name":"BYOL"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.06810","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.06810"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miszkur/SelfSupervisedLearning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/byol-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fmi-basel/implicit-var-reg","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/luckmatters","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/luckmatters/tree/master/ssl","reach":null}],"summary":{"ran":1,"ran_violates":3,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"551a048173369c49","entry":"Accumulator","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"551a048173369c49"}},{"code_sha256_prefix":"1c0f269ae9673b08","entry":"default","repo":"lucidrains/byol-pytorch","repo_kind":"listed","path":"byol_pytorch/byol_pytorch.py","file_url":"https://github.com/lucidrains/byol-pytorch/blob/HEAD/byol_pytorch/byol_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c0f269ae9673b08"}},{"code_sha256_prefix":"f912c209e9c899ca","entry":"divisible_by","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f912c209e9c899ca"}},{"code_sha256_prefix":"b7c3487f192e31b7","entry":"exists","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b7c3487f192e31b7"}},{"code_sha256_prefix":"418a6e830639c7e1","entry":"random_orthonormal_matrix","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"418a6e830639c7e1"}},{"code_sha256_prefix":"a25c0f4514f8e550","entry":"BYOLTrainer","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a25c0f4514f8e550"}},{"code_sha256_prefix":"5b53f8127f8ee5f7","entry":"_create_model_training_folder","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5b53f8127f8ee5f7"}},{"code_sha256_prefix":"34e6134013a7af6e","entry":"check_valid","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"34e6134013a7af6e"}},{"code_sha256_prefix":"f9ada57830668708","entry":"weights_init","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"ssl/real-dataset/byol_trainer.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/ssl/real-dataset/byol_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f9ada57830668708"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}