{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-distilled-self-supervised-representation","title":"Self-Distilled Self-Supervised Representation Learning","arxiv_id":"2111.12958","date":"2021-11-25","proceeding":null,"authors":["Jiho Jang","Seonhoon Kim","KiYoon Yoo","Chaerin Kong","Jangho Kim","Nojun Kwak"],"abstract":"State-of-the-art frameworks in self-supervised learning have recently shown that fully utilizing transformer-based models can lead to performance boost compared to conventional CNN models. Striving to maximize the mutual information of two views of an image, existing works apply a contrastive loss to the final representations. Motivated by self-distillation in the supervised regime, we further exploit this by allowing the intermediate representations to learn from the final layer via the contrastive loss. Through self-distillation, the intermediate layers are better suited for instance discrimination, making the performance of an early-exited sub-network not much degraded from that of the full network. This renders the pretext task easier also for the final layer, leading to better representations. Our method, Self-Distilled Self-Supervised Learning (SDSSL), outperforms competitive baselines (SimCLR, BYOL and MoCo v3) using ViT on various tasks and datasets. In the linear evaluation and k-NN protocol, SDSSL not only leads to superior performance in the final layers, but also in most of the lower layers. Furthermore, qualitative and quantitative analyses show how representations are formed more effectively along the transformer layers. Code is available at https://github.com/hagiss/SDSSL.","url_abs":"https://arxiv.org/abs/2111.12958v3","url_pdf":"https://arxiv.org/pdf/2111.12958v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-distilled-self-supervised-representation","repo_url":"https://github.com/hagiss/sdssl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"byol","method_name":"BYOL"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"moco","method_name":"MoCo"},{"method_slug":"k-nn","method_name":"k-NN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.12958","atlas_url":"https://app.syntology.ai/?focus=2111.12958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hagiss/sdssl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_honours":1,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c19231378e41dc5a","entry":"bool_flag","repo":"hagiss/sdssl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c19231378e41dc5a"}},{"code_sha256_prefix":"2b5489053b222e00","entry":"clip_gradients","repo":"hagiss/sdssl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5489053b222e00"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"hagiss/sdssl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"1c0f269ae9673b08","entry":"default","repo":"hagiss/sdssl","repo_kind":"official","path":"pl_train_byol.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/pl_train_byol.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c0f269ae9673b08"}},{"code_sha256_prefix":"55120f2026b56aa2","entry":"drop_path","repo":"hagiss/sdssl","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55120f2026b56aa2"}},{"code_sha256_prefix":"00cae89471470576","entry":"loss_fn","repo":"hagiss/sdssl","repo_kind":"official","path":"pl_train_byol.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/pl_train_byol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00cae89471470576"}},{"code_sha256_prefix":"e53d0d8dc48485af","entry":"loss_fn","repo":"hagiss/sdssl","repo_kind":"official","path":"pl_train_moco.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/pl_train_moco.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e53d0d8dc48485af"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"hagiss/sdssl","repo_kind":"official","path":"pl_train_simclr.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/pl_train_simclr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"ff3be2bd472acb68","entry":"count_parameters","repo":"hagiss/sdssl","repo_kind":"official","path":"pl_train_byol.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/pl_train_byol.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff3be2bd472acb68"}},{"code_sha256_prefix":"e00c06f711a4e207","entry":"vit_small","repo":"hagiss/sdssl","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e00c06f711a4e207"}},{"code_sha256_prefix":"467133f2b3c000e2","entry":"vit_tiny","repo":"hagiss/sdssl","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/hagiss/sdssl/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"467133f2b3c000e2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}