{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-damaging-contrastive-learning","title":"Self-Damaging Contrastive Learning","arxiv_id":"2106.02990","date":"2021-06-06","proceeding":null,"authors":["Ziyu Jiang","Tianlong Chen","Bobak Mortazavi","Zhangyang Wang"],"abstract":"The recent breakthrough achieved by contrastive learning accelerates the pace for deploying unsupervised training on real-world data applications. However, unlabeled data in reality is commonly imbalanced and shows a long-tail distribution, and it is unclear how robustly the latest contrastive learning methods could perform in the practical scenario. This paper proposes to explicitly tackle this challenge, via a principled framework called Self-Damaging Contrastive Learning (SDCLR), to automatically balance the representation learning without knowing the classes. Our main inspiration is drawn from the recent finding that deep models have difficult-to-memorize samples, and those may be exposed through network pruning. It is further natural to hypothesize that long-tail samples are also tougher for the model to learn well due to insufficient examples. Hence, the key innovation in SDCLR is to create a dynamic self-competitor model to contrast with the target model, which is a pruned version of the latter. During training, contrasting the two models will lead to adaptive online mining of the most easily forgotten samples for the current target model, and implicitly emphasize them more in the contrastive loss. Extensive experiments across multiple datasets and imbalance settings show that SDCLR significantly improves not only overall accuracies but also balancedness, in terms of linear evaluation on the full-shot and few-shot settings. Our code is available at: https://github.com/VITA-Group/SDCLR.","url_abs":"https://arxiv.org/abs/2106.02990v1","url_pdf":"https://arxiv.org/pdf/2106.02990v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-damaging-contrastive-learning","repo_url":"https://github.com/VITA-Group/SDCLR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.02990","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.02990"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/VITA-Group/SDCLR","reach":null}],"summary":{"ran":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"b98e4f9cea01e67d","entry":"batch_norm_multiple","repo":"VITA-Group/SDCLR","repo_kind":"official","path":"models/resnet_prune_multibn.py","file_url":"https://github.com/VITA-Group/SDCLR/blob/HEAD/models/resnet_prune_multibn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b98e4f9cea01e67d"}},{"code_sha256_prefix":"00ff99a4d89b9134","entry":"getStatisticsFromTxt","repo":"VITA-Group/SDCLR","repo_kind":"official","path":"exp_analyse.py","file_url":"https://github.com/VITA-Group/SDCLR/blob/HEAD/exp_analyse.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00ff99a4d89b9134"}},{"code_sha256_prefix":"77f767f22442b9b2","entry":"proj_head","repo":"VITA-Group/SDCLR","repo_kind":"official","path":"models/resnet_prune_multibn.py","file_url":"https://github.com/VITA-Group/SDCLR/blob/HEAD/models/resnet_prune_multibn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77f767f22442b9b2"}},{"code_sha256_prefix":"cfae83d8747a33da","entry":"cvt_state_dict","repo":"VITA-Group/SDCLR","repo_kind":"official","path":"train_cifar.py","file_url":"https://github.com/VITA-Group/SDCLR/blob/HEAD/train_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfae83d8747a33da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}