{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/better-supervisory-signals-by-observing-1","title":"Better Supervisory Signals by Observing Learning Paths","arxiv_id":"2203.02485","date":"2022-03-04","proceeding":"ICLR 2022 4","authors":["Yi Ren","Shangmin Guo","Danica J. Sutherland"],"abstract":"Better-supervised models might have better performance. In this paper, we first clarify what makes for good supervision for a classification problem, and then explain two existing label refining methods, label smoothing and knowledge distillation, in terms of our proposed criterion. To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the model's predictions during training, for each training sample. We find that the model can spontaneously refine \"bad\" labels through a \"zig-zag\" learning path, which occurs on both toy and real datasets. Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks. Inspired by this, we propose a new knowledge distillation scheme, Filter-KD, which improves downstream classification performance in various settings.","url_abs":"https://arxiv.org/abs/2203.02485v1","url_pdf":"https://arxiv.org/pdf/2203.02485v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"better-supervisory-signals-by-observing-1","repo_url":"https://github.com/joshua-ren/better_supervisory_signal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.02485","atlas_url":"https://app.syntology.ai/?focus=2203.02485","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.02485"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Joshua-Ren/better_supervisory_signal","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joshua-ren/better_supervisory_signal","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_violates":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"21fe6fa61f6e0652","entry":"ce_loss","repo":"joshua-ren/better_supervisory_signal","repo_kind":"official","path":"utils.py","file_url":"https://github.com/joshua-ren/better_supervisory_signal/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21fe6fa61f6e0652"}},{"code_sha256_prefix":"da57ff4737709565","entry":"distil_loss","repo":"joshua-ren/better_supervisory_signal","repo_kind":"official","path":"utils.py","file_url":"https://github.com/joshua-ren/better_supervisory_signal/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da57ff4737709565"}},{"code_sha256_prefix":"fdceb841db6be746","entry":"get_validation","repo":"joshua-ren/better_supervisory_signal","repo_kind":"official","path":"main_distill.py","file_url":"https://github.com/joshua-ren/better_supervisory_signal/blob/HEAD/main_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdceb841db6be746"}},{"code_sha256_prefix":"741d6b3c4d0906d4","entry":"kd_loss","repo":"joshua-ren/better_supervisory_signal","repo_kind":"official","path":"utils.py","file_url":"https://github.com/joshua-ren/better_supervisory_signal/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"741d6b3c4d0906d4"}},{"code_sha256_prefix":"edb82c7ef88ade05","entry":"get_init_net","repo":"joshua-ren/better_supervisory_signal","repo_kind":"official","path":"utils.py","file_url":"https://github.com/joshua-ren/better_supervisory_signal/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edb82c7ef88ade05"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}