{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/solving-inefficiency-of-self-supervised","title":"Solving Inefficiency of Self-supervised Representation Learning","arxiv_id":"2104.08760","date":"2021-04-18","proceeding":"ICCV 2021 10","authors":["Guangrun Wang","Keze Wang","Guangcong Wang","Philip H. S. Torr","Liang Lin"],"abstract":"Self-supervised learning (especially contrastive learning) has attracted great interest due to its huge potential in learning discriminative representations in an unsupervised manner. Despite the acknowledged successes, existing contrastive learning methods suffer from very low learning efficiency, e.g., taking about ten times more training epochs than supervised learning for comparable recognition accuracy. In this paper, we reveal two contradictory phenomena in contrastive learning that we call under-clustering and over-clustering problems, which are major obstacles to learning efficiency. Under-clustering means that the model cannot efficiently learn to discover the dissimilarity between inter-class samples when the negative sample pairs for contrastive learning are insufficient to differentiate all the actual object classes. Over-clustering implies that the model cannot efficiently learn features from excessive negative sample pairs, forcing the model to over-cluster samples of the same actual classes into different clusters. To simultaneously overcome these two problems, we propose a novel self-supervised learning framework using a truncated triplet loss. Precisely, we employ a triplet loss tending to maximize the relative distance between the positive pair and negative pairs to address the under-clustering problem; and we construct the negative pair by selecting a negative sample deputy from all negative samples to avoid the over-clustering problem, guaranteed by the Bernoulli Distribution model. We extensively evaluate our framework in several large-scale benchmarks (e.g., ImageNet, SYSU-30k, and COCO). The results demonstrate our model's superiority (e.g., the learning efficiency) over the latest state-of-the-art methods by a clear margin. Codes available at: https://github.com/wanggrun/triplet .","url_abs":"https://arxiv.org/abs/2104.08760v3","url_pdf":"https://arxiv.org/pdf/2104.08760v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"solving-inefficiency-of-self-supervised","repo_url":"https://github.com/wanggrun/triplet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"person-re-identification","task_name":"Person Re-Identification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"self-supervised-person-re-identification","task_name":"Self-Supervised Person Re-Identification"},{"task_slug":null,"task_name":"Triplet"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"triplet-loss","method_name":"Triplet Loss"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/person-re-identification-on-sysu-30k","task":"Person Re-Identification","dataset":"SYSU-30k","model":"Triplet (self-supervised)","rank_in_archive_order":4,"of":10,"metrics":{" Rank-1":"14.8"},"uses_additional_data":true},{"leaderboard":"/sota/self-supervised-image-classification-on","task":"Self-Supervised Image Classification","dataset":"ImageNet","model":"Triplet  (ResNet-50)","rank_in_archive_order":65,"of":144,"metrics":{"Number of Params":"23.56M","Top 1 Accuracy":"75.9%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-person-re-identification-on","task":"Self-Supervised Person Re-Identification","dataset":"SYSU-30k","model":"Triplet","rank_in_archive_order":1,"of":4,"metrics":{" Rank-1":"14.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08760","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08760"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wanggrun/triplet","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5456c207a88ac560","entry":"Registry","repo":"wanggrun/triplet","repo_kind":"official","path":"openselfsup/models/heads/triplet_loss_head.py","file_url":"https://github.com/wanggrun/triplet/blob/HEAD/openselfsup/models/heads/triplet_loss_head.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5456c207a88ac560"}},{"code_sha256_prefix":"da3963a12daedd3a","entry":"TripletLossHead","repo":"wanggrun/triplet","repo_kind":"official","path":"openselfsup/models/heads/triplet_loss_head.py","file_url":"https://github.com/wanggrun/triplet/blob/HEAD/openselfsup/models/heads/triplet_loss_head.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da3963a12daedd3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}