{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cluster-aware-semi-supervised-learning","title":"Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering","arxiv_id":"2307.11030","date":"2023-07-20","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the corresponding theoretical interpretations remain limited for various knowledge distillation paradigms. In this work, we take an initial step toward a theoretical understanding of relational knowledge distillation (RKD), with a focus on semi-supervised classification problems. We start by casting RKD as spectral clustering on a population-induced graph unveiled by a teacher model. Via a notion of clustering error that quantifies the discrepancy between the predicted and ground truth clusterings, we illustrate that RKD over the population provably leads to low clustering error. Moreover, we provide a sample complexity bound for RKD with limited unlabeled samples. For semi-supervised learning, we further demonstrate the label efficiency of RKD through a general framework of cluster-aware semi-supervised learning that assumes low clustering errors. Finally, by unifying data augmentation consistency regularization into this cluster-aware framework, we show that despite the common effect of learning accurate clusterings, RKD facilitates a \"global\" perspective through spectral clustering, whereas consistency regularization focuses on a \"local\" perspective via expansion.","url_abs":"https://arxiv.org/abs/2307.11030v2","url_pdf":"https://arxiv.org/pdf/2307.11030v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cluster-aware-semi-supervised-learning","repo_url":"https://github.com/dyjdongyijun/semi_supervised_knowledge_distillation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"spectral-clustering","method_name":"Spectral Clustering"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.11030","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.11030"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dyjdongyijun/semi_supervised_knowledge_distillation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":14,"ran_fixture":1,"unverified":3},"by_repo_kind":{"listed":{"samples":18,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c6e7c3451c85404f","entry":"AutoContrast","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/randaugment.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6e7c3451c85404f"}},{"code_sha256_prefix":"3009787d942b1c5c","entry":"BasicBlock","repo":"dyjdongyijun/semi_supervised_knowledge_distillation","repo_kind":"listed","path":"models/wideresnet.py","file_url":"https://github.com/dyjdongyijun/semi_supervised_knowledge_distillation/blob/HEAD/models/wideresnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3009787d942b1c5c"}},{"code_sha256_prefix":"0d1a8d6ce2015fc7","entry":"Brightness","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/randaugment.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d1a8d6ce2015fc7"}},{"code_sha256_prefix":"32cb3f5760fa593e","entry":"Color","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/randaugment.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32cb3f5760fa593e"}},{"code_sha256_prefix":"c4636e780898f2ee","entry":"NetworkBlock","repo":"dyjdongyijun/semi_supervised_knowledge_distillation","repo_kind":"listed","path":"models/wideresnet.py","file_url":"https://github.com/dyjdongyijun/semi_supervised_knowledge_distillation/blob/HEAD/models/wideresnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4636e780898f2ee"}},{"code_sha256_prefix":"4c81050862db0aaa","entry":"WideResNet","repo":"dyjdongyijun/semi_supervised_knowledge_distillation","repo_kind":"listed","path":"models/wideresnet.py","file_url":"https://github.com/dyjdongyijun/semi_supervised_knowledge_distillation/blob/HEAD/models/wideresnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c81050862db0aaa"}},{"code_sha256_prefix":"18badd533b52adb3","entry":"accuracy","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"utils/misc.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18badd533b52adb3"}},{"code_sha256_prefix":"fa7ed453d0ada3a4","entry":"get_cifar100_config","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"teach.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/teach.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa7ed453d0ada3a4"}},{"code_sha256_prefix":"3caa7c7dc0755584","entry":"get_cifar10_config","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"teach.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/teach.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3caa7c7dc0755584"}},{"code_sha256_prefix":"8c82d522e7c85b05","entry":"get_mean_and_std","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"utils/misc.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c82d522e7c85b05"}},{"code_sha256_prefix":"0c89b557d9797e74","entry":"get_offline_teacher","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/cifar.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c89b557d9797e74"}},{"code_sha256_prefix":"f499063bfc9fc8b4","entry":"get_timagenet_config","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"teach.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/teach.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f499063bfc9fc8b4"}},{"code_sha256_prefix":"0a7c46eb67c0c21b","entry":"mish","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"models/wideresnet.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/models/wideresnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a7c46eb67c0c21b"}},{"code_sha256_prefix":"f100d3606ccfdf9b","entry":"train_val_transforms","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/cifar.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f100d3606ccfdf9b"}},{"code_sha256_prefix":"2cb92fa08ea17769","entry":"x_u_split","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/fashionmnist.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/fashionmnist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cb92fa08ea17769"}},{"code_sha256_prefix":"73e41df974d19846","entry":"build_wideresnet","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"models/wideresnet.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/models/wideresnet.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73e41df974d19846"}},{"code_sha256_prefix":"9758b8f7ba37ba52","entry":"get_fashionmnist","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/fashionmnist.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/fashionmnist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9758b8f7ba37ba52"}},{"code_sha256_prefix":"746f59a1a4bb48a4","entry":"get_tiny_imagenet","repo":"dyjdongyijun/Semi_Supervised_Knowledge_Distillation","repo_kind":"listed","path":"dataset/tiny_imagenet.py","file_url":"https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation/blob/HEAD/dataset/tiny_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"746f59a1a4bb48a4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}