{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-collapse-in-non-contrastive","title":"Understanding Collapse in Non-Contrastive Siamese Representation Learning","arxiv_id":"2209.15007","date":"2022-09-29","proceeding":null,"authors":["Alexander C. Li","Alexei A. Efros","Deepak Pathak"],"abstract":"Contrastive methods have led a recent surge in the performance of self-supervised representation learning (SSL). Recent methods like BYOL or SimSiam purportedly distill these contrastive methods down to their essence, removing bells and whistles, including the negative examples, that do not contribute to downstream performance. These \"non-contrastive\" methods work surprisingly well without using negatives even though the global minimum lies at trivial collapse. We empirically analyze these non-contrastive methods and find that SimSiam is extraordinarily sensitive to dataset and model size. In particular, SimSiam representations undergo partial dimensional collapse if the model is too small relative to the dataset size. We propose a metric to measure the degree of this collapse and show that it can be used to forecast the downstream task performance without any fine-tuning or labels. We further analyze architectural design choices and their effect on the downstream performance. Finally, we demonstrate that shifting to a continual learning setting acts as a regularizer and prevents collapse, and a hybrid between continual and multi-epoch training can improve linear probe accuracy by as many as 18 percentage points using ResNet-18 on ImageNet. Our project page is at https://alexanderli.com/noncontrastive-ssl/.","url_abs":"https://arxiv.org/abs/2209.15007v2","url_pdf":"https://arxiv.org/pdf/2209.15007v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-collapse-in-non-contrastive","repo_url":"https://github.com/alexlioralexli/noncontrastive-ssl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"byol","method_name":"BYOL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.15007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.15007"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alexlioralexli/noncontrastive-ssl","reach":null}],"summary":{"ran_fixture":1,"ran":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"18e389acb9d4afdb","entry":"accuracy","repo":"alexlioralexli/noncontrastive-ssl","repo_kind":"official","path":"main_lincls.py","file_url":"https://github.com/alexlioralexli/noncontrastive-ssl/blob/HEAD/main_lincls.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18e389acb9d4afdb"}},{"code_sha256_prefix":"0f978bc80f0a89f8","entry":"get_incremental_dataloader","repo":"alexlioralexli/noncontrastive-ssl","repo_kind":"official","path":"main_simsiam.py","file_url":"https://github.com/alexlioralexli/noncontrastive-ssl/blob/HEAD/main_simsiam.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f978bc80f0a89f8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}