{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-learning-is-more-robust-to-1","title":"Self-supervised Learning is More Robust to Dataset Imbalance","arxiv_id":"2110.05025","date":"2021-10-11","proceeding":"ICLR 2022 4","authors":["Hong Liu","Jeff Z. HaoChen","Adrien Gaidon","Tengyu Ma"],"abstract":"Self-supervised learning (SSL) is a scalable way to learn general visual representations since it learns without labels. However, large-scale unlabeled datasets in the wild often have long-tailed label distributions, where we know little about the behavior of SSL. In this work, we systematically investigate self-supervised learning under dataset imbalance. First, we find out via extensive experiments that off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. The performance gap between balanced and imbalanced pre-training with SSL is significantly smaller than the gap with supervised learning, across sample sizes, for both in-domain and, especially, out-of-domain evaluation. Second, towards understanding the robustness of SSL, we hypothesize that SSL learns richer features from frequent data: it may learn label-irrelevant-but-transferable features that help classify the rare classes and downstream tasks. In contrast, supervised learning has no incentive to learn features irrelevant to the labels from frequent examples. We validate this hypothesis with semi-synthetic experiments and theoretical analyses on a simplified setting. Third, inspired by the theoretical insights, we devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets with several evaluation criteria, closing the small gap between balanced and imbalanced datasets with the same number of examples.","url_abs":"https://arxiv.org/abs/2110.05025v2","url_pdf":"https://arxiv.org/pdf/2110.05025v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-is-more-robust-to-1","repo_url":"https://github.com/Liuhong99/Imbalanced-SSL","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"long-tail-learning","task_name":"Long-tail Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/long-tail-learning-on-cifar-10-lt-r-100","task":"Long-tail Learning","dataset":"CIFAR-10-LT (ρ=100)","model":"SimSiam+rwSAM","rank_in_archive_order":10,"of":28,"metrics":{"Error Rate":"14.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.05025","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.05025"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Liuhong99/Imbalanced-SSL","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"named_in_paper":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"b58f7b3f0347a073","entry":"CrossEntropyLoss1","repo":"Liuhong99/Imbalanced-SSL","repo_kind":"named_in_paper","path":"imagenet/pretrain_moco_sam_rare.py","file_url":"https://github.com/Liuhong99/Imbalanced-SSL/blob/HEAD/imagenet/pretrain_moco_sam_rare.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b58f7b3f0347a073"}},{"code_sha256_prefix":"983576fa67f4f01e","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"983576fa67f4f01e"}},{"code_sha256_prefix":"ee45897c218040d1","entry":"train","repo":"Liuhong99/Imbalanced-SSL","repo_kind":"named_in_paper","path":"cifar/main_sam_exp.py","file_url":"https://github.com/Liuhong99/Imbalanced-SSL/blob/HEAD/cifar/main_sam_exp.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee45897c218040d1"}},{"code_sha256_prefix":"715aadc8b870f357","entry":"load_checkpoint","repo":"Liuhong99/Imbalanced-SSL","repo_kind":"named_in_paper","path":"cifar/main_sam_exp.py","file_url":"https://github.com/Liuhong99/Imbalanced-SSL/blob/HEAD/cifar/main_sam_exp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"715aadc8b870f357"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}