{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-a-student-becomes-a-teacher-learning-and","title":"How a student becomes a teacher: learning and forgetting through Spectral methods","arxiv_id":"2310.12612","date":"2023-10-19","proceeding":"NeurIPS 2023 11","authors":["Lorenzo Giambagli","Lorenzo Buffoni","Lorenzo Chicchi","Duccio Fanelli"],"abstract":"In theoretical ML, the teacher-student paradigm is often employed as an effective metaphor for real-life tuition. The above scheme proves particularly relevant when the student network is overparameterized as compared to the teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits.","url_abs":"https://arxiv.org/abs/2310.12612v2","url_pdf":"https://arxiv.org/pdf/2310.12612v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-a-student-becomes-a-teacher-learning-and","repo_url":"https://github.com/jamba15/spectral-regularization-teacher-student","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.12612","atlas_url":"https://app.syntology.ai/?focus=2310.12612","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12612"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Jamba15/SpectralTools","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Jamba15/Spectral-regularization-teacher-student","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jamba15/spectral-regularization-teacher-student","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"ef484f9c68029f71","entry":"DictionaryReplace","repo":"Jamba15/Spectral-regularization-teacher-student","repo_kind":"official","path":"proj_functions.py","file_url":"https://github.com/Jamba15/Spectral-regularization-teacher-student/blob/HEAD/proj_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef484f9c68029f71"}},{"code_sha256_prefix":"1831c7fe2bd4f000","entry":"LinearTeacher","repo":"Jamba15/Spectral-regularization-teacher-student","repo_kind":"official","path":"proj_functions.py","file_url":"https://github.com/Jamba15/Spectral-regularization-teacher-student/blob/HEAD/proj_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1831c7fe2bd4f000"}},{"code_sha256_prefix":"30d12bc306c98764","entry":"get_layer","repo":"Jamba15/SpectralTools","repo_kind":"found_in_text","path":"PyTorch/utils.py","file_url":"https://github.com/Jamba15/SpectralTools/blob/HEAD/PyTorch/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d12bc306c98764"}},{"code_sha256_prefix":"fe3546546e64b493","entry":"LoadConfig","repo":"Jamba15/Spectral-regularization-teacher-student","repo_kind":"official","path":"proj_functions.py","file_url":"https://github.com/Jamba15/Spectral-regularization-teacher-student/blob/HEAD/proj_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe3546546e64b493"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}