{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-sustainable-self-supervised-learning","title":"Towards Sustainable Self-supervised Learning","arxiv_id":"2210.11016","date":"2022-10-20","proceeding":null,"authors":["ShangHua Gao","Pan Zhou","Ming-Ming Cheng","Shuicheng Yan"],"abstract":"Although increasingly training-expensive, most self-supervised learning (SSL) models have repeatedly been trained from scratch but not fully utilized, since only a few SOTAs are employed for downstream tasks. In this work, we explore a sustainable SSL framework with two major challenges: i) learning a stronger new SSL model based on the existing pretrained SSL model, also called as \"base\" model, in a cost-friendly manner, ii) allowing the training of the new model to be compatible with various base models. We propose a Target-Enhanced Conditional (TEC) scheme which introduces two components to the existing mask-reconstruction based SSL. Firstly, we propose patch-relation enhanced targets which enhances the target given by base model and encourages the new model to learn semantic-relation knowledge from the base model by using incomplete inputs. This hardening and target-enhancing help the new model surpass the base model, since they enforce additional patch relation modeling to handle incomplete input. Secondly, we introduce a conditional adapter that adaptively adjusts new model prediction to align with the target of different base models. Extensive experimental results show that our TEC scheme can accelerate the learning speed, and also improve SOTA SSL base models, e.g., MAE and iBOT, taking an explorative step towards sustainable SSL.","url_abs":"https://arxiv.org/abs/2210.11016v1","url_pdf":"https://arxiv.org/pdf/2210.11016v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-sustainable-self-supervised-learning","repo_url":"https://github.com/sail-sg/tec","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"mae","method_name":"MAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"TEC(VIT-B, Mask-RCNN)","rank_in_archive_order":54,"of":220,"metrics":{"box AP":"54.6"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"TEC_MAE (ViT-L/16, 224)","rank_in_archive_order":13,"of":65,"metrics":{"Top 1 Accuracy":"86.5%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"TEC (Vit-B, Upernet)","rank_in_archive_order":104,"of":235,"metrics":{"Validation mIoU":"51.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-imagenet-s","task":"Semantic Segmentation","dataset":"ImageNet-S","model":"TEC (ViT-B/16, 224x224, SSL+FT, mmseg)","rank_in_archive_order":1,"of":20,"metrics":{"mIoU (test)":"62.5","mIoU (val)":"63.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-imagenet-s","task":"Semantic Segmentation","dataset":"ImageNet-S","model":"TEC (ViT-B/16, 224x224, SSL+FT)","rank_in_archive_order":3,"of":20,"metrics":{"mIoU (val)":"62.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-imagenet-s","task":"Semantic Segmentation","dataset":"ImageNet-S","model":"TEC (ViT-B/16, 224x224, SSL, mmseg)","rank_in_archive_order":13,"of":20,"metrics":{"mIoU (test)":"46.0","mIoU (val)":"46.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-imagenet-s","task":"Semantic Segmentation","dataset":"ImageNet-S","model":"TEC (ViT-B/16, 224x224, SSL)","rank_in_archive_order":14,"of":20,"metrics":{"mIoU (val)":"42.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.11016","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.11016"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/tec","reach":null}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"bde9fdb88d04b910","entry":"Attention","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bde9fdb88d04b910"}},{"code_sha256_prefix":"18847b8fadd70db0","entry":"Block","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18847b8fadd70db0"}},{"code_sha256_prefix":"8db0a24fc6a85720","entry":"FeatureAdaptor","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8db0a24fc6a85720"}},{"code_sha256_prefix":"f58e26de5060150d","entry":"PatchEmbed","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f58e26de5060150d"}},{"code_sha256_prefix":"19e9319d6e88e674","entry":"ViTDecoderAtt","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19e9319d6e88e674"}},{"code_sha256_prefix":"6a4f060bf556b84d","entry":"ViTDecoderFeature","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a4f060bf556b84d"}},{"code_sha256_prefix":"fbf181dd5250dedb","entry":"TECViT","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbf181dd5250dedb"}},{"code_sha256_prefix":"7089e8a76f671a0f","entry":"get_2d_sincos_pos_embed","repo":"sail-sg/tec","repo_kind":"official","path":"models/models_tec_vit.py","file_url":"https://github.com/sail-sg/tec/blob/HEAD/models/models_tec_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7089e8a76f671a0f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}