{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-learning-by-estimating-twin-1","title":"Self-Supervised Learning by Estimating Twin Class Distributions","arxiv_id":"2110.07402","date":"2021-10-14","proceeding":null,"authors":["Feng Wang","Tao Kong","Rufeng Zhang","Huaping Liu","Hang Li"],"abstract":"We present TWIST, a simple and theoretically explainable self-supervised representation learning method by classifying large-scale unlabeled datasets in an end-to-end way. We employ a siamese network terminated by a softmax operation to produce twin class distributions of two augmented images. Without supervision, we enforce the class distributions of different augmentations to be consistent. However, simply minimizing the divergence between augmentations will cause collapsed solutions, i.e., outputting the same class probability distribution for all images. In this case, no information about the input image is left. To solve this problem, we propose to maximize the mutual information between the input and the class predictions. Specifically, we minimize the entropy of the distribution for each sample to make the class prediction for each sample assertive and maximize the entropy of the mean distribution to make the predictions of different samples diverse. In this way, TWIST can naturally avoid the collapsed solutions without specific designs such as asymmetric network, stop-gradient operation, or momentum encoder. As a result, TWIST outperforms state-of-the-art methods on a wide range of tasks. Especially, TWIST performs surprisingly well on semi-supervised learning, achieving 61.2% top-1 accuracy with 1% ImageNet labels using a ResNet-50 as backbone, surpassing previous best results by an absolute improvement of 6.2%. Codes and pre-trained models are given on: https://github.com/bytedance/TWIST","url_abs":"https://arxiv.org/abs/2110.07402v4","url_pdf":"https://arxiv.org/pdf/2110.07402v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-by-estimating-twin-1","repo_url":"https://github.com/bytedance/TWIST","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"self-supervised-learning-by-estimating-twin-1","repo_url":"https://github.com/beresandras/contrastive-classification-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"fine-grained-image-classification","task_name":"Fine-Grained Image Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semi-supervised-image-classification","task_name":"Semi-Supervised Image Classification"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"unsupervised-image-classification","task_name":"Unsupervised Image Classification"}],"methods":[{"method_slug":"siamese-network","method_name":"Siamese Network"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/fine-grained-image-classification-on-caltech","task":"Fine-Grained Image Classification","dataset":"Caltech-101","model":"TWIST (ResNet-50 )","rank_in_archive_order":6,"of":18,"metrics":{"Accuracy":"93.5%","Top-1 Error Rate":"6.5%"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-image-classification-on-sun397","task":"Fine-Grained Image Classification","dataset":"SUN397","model":"TWIST (ResNet-50)","rank_in_archive_order":4,"of":5,"metrics":{"Accuracy":"67.4"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-dtd","task":"Image Classification","dataset":"DTD","model":"TWIST (ResNet-50)","rank_in_archive_order":8,"of":11,"metrics":{"Accuracy":"76.6"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-food-101-1","task":"Image Classification","dataset":"Food-101","model":"TWIST (ResNet-50)","rank_in_archive_order":5,"of":11,"metrics":{"Accuracy (%)":"89.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets","task":"Image Classification","dataset":"Oxford-IIIT Pet Dataset","model":"TWIST (ResNet-50)","rank_in_archive_order":1,"of":5,"metrics":{"Accuracy":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/semi-supervised-image-classification-on-1","task":"Semi-Supervised Image Classification","dataset":"ImageNet - 1% labeled data","model":"TWIST (ResNet-50 x2)","rank_in_archive_order":26,"of":65,"metrics":{"Top 1 Accuracy":"67.2%","Top 5 Accuracy":"88.2%"},"uses_additional_data":false},{"leaderboard":"/sota/semi-supervised-image-classification-on-2","task":"Semi-Supervised Image Classification","dataset":"ImageNet - 10% labeled data","model":"TWIST (ResNet-50 x2)","rank_in_archive_order":26,"of":75,"metrics":{"Top 1 Accuracy":"75.3%","Top 5 Accuracy":"92.8%"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-image-classification-on-imagenet","task":"Unsupervised Image Classification","dataset":"ImageNet","model":"TWIST (ResNet-50)","rank_in_archive_order":6,"of":9,"metrics":{"ARI":"30.0","Accuracy (%)":"40.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.07402","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.07402"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/beresandras/contrastive-classification-keras","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytedance/TWIST","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":2,"ran_honours":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b0f936d4d6ae3b8c","entry":"accuracy","repo":"bytedance/TWIST","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0f936d4d6ae3b8c"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"bytedance/TWIST","repo_kind":"official","path":"widen_resnet.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/widen_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"bytedance/TWIST","repo_kind":"official","path":"widen_resnet.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/widen_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"bytedance/TWIST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"915b71e73a54fc97","entry":"trunc_normal_","repo":"bytedance/TWIST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"915b71e73a54fc97"}},{"code_sha256_prefix":"5067fd3e4ebe3111","entry":"CE","repo":"bytedance/TWIST","repo_kind":"official","path":"objective.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/objective.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5067fd3e4ebe3111"}},{"code_sha256_prefix":"a74d130de19016d6","entry":"KL","repo":"bytedance/TWIST","repo_kind":"official","path":"objective.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/objective.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a74d130de19016d6"}},{"code_sha256_prefix":"f6d031f1affc8b74","entry":"calculate_cost_matrix","repo":"bytedance/TWIST","repo_kind":"official","path":"evaluate_cluster.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/evaluate_cluster.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6d031f1affc8b74"}},{"code_sha256_prefix":"8b7b49496df1bbe5","entry":"check_path","repo":"bytedance/TWIST","repo_kind":"official","path":"imagenet_lmdb.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/imagenet_lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b7b49496df1bbe5"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"bytedance/TWIST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"e3880dee498de475","entry":"evaluate","repo":"bytedance/TWIST","repo_kind":"official","path":"evaluate_cluster.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/evaluate_cluster.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3880dee498de475"}},{"code_sha256_prefix":"f8fa05939509a76b","entry":"get_augmentations","repo":"bytedance/TWIST","repo_kind":"official","path":"augmentation.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/augmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8fa05939509a76b"}},{"code_sha256_prefix":"c5807f4f8acc5c0b","entry":"get_cluster_labels_from_indices","repo":"bytedance/TWIST","repo_kind":"official","path":"evaluate_cluster.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/evaluate_cluster.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5807f4f8acc5c0b"}},{"code_sha256_prefix":"b4c154b6f0568b6b","entry":"loads_pyarrow","repo":"bytedance/TWIST","repo_kind":"official","path":"imagenet_lmdb.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/imagenet_lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4c154b6f0568b6b"}},{"code_sha256_prefix":"d2772e0a588e4529","entry":"resnet18","repo":"bytedance/TWIST","repo_kind":"official","path":"widen_resnet.py","file_url":"https://github.com/bytedance/TWIST/blob/HEAD/widen_resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2772e0a588e4529"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}