{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-one-pass-self-distillation-with","title":"Efficient One Pass Self-distillation with Zipf's Label Smoothing","arxiv_id":"2207.12980","date":"2022-07-26","proceeding":null,"authors":["Jiajun Liang","Linze Li","Zhaodong Bing","Borui Zhao","Yao Tang","Bo Lin","Haoqiang Fan"],"abstract":"Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is often overlooked, and yet reducing time and memory overhead during training is increasingly important in the giant models' era. This paper proposes an efficient self-distillation method named Zipf's Label Smoothing (Zipf's LS), which uses the on-the-fly prediction of a network to generate soft supervision that conforms to Zipf distribution without using any contrastive samples or auxiliary parameters. Our idea comes from an empirical observation that when the network is duly trained the output values of a network's final softmax layer, after sorting by the magnitude and averaged across samples, should follow a distribution reminiscent to Zipf's Law in the word frequency statistics of natural languages. By enforcing this property on the sample level and throughout the whole training period, we find that the prediction accuracy can be greatly improved. Using ResNet50 on the INAT21 fine-grained classification dataset, our technique achieves +3.61% accuracy gain compared to the vanilla baseline, and 0.88% more gain against the previous label smoothing or self-distillation strategies. The implementation is publicly available at https://github.com/megvii-research/zipfls.","url_abs":"https://arxiv.org/abs/2207.12980v1","url_pdf":"https://arxiv.org/pdf/2207.12980v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-one-pass-self-distillation-with","repo_url":"https://github.com/megvii-research/zipfls","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.12980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.12980"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/megvii-research/zipfls","reach":null}],"summary":{"ran_fixture":3,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a1377a500e1d96b1","entry":"gen_pdf","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1377a500e1d96b1"}},{"code_sha256_prefix":"37891eae608fb0e6","entry":"gen_pdf_linear_decay","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37891eae608fb0e6"}},{"code_sha256_prefix":"2b4cafb118bb8090","entry":"gen_pdf_unbiased","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b4cafb118bb8090"}},{"code_sha256_prefix":"8acc4e763cd2a6c1","entry":"get_masked_distribution","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8acc4e763cd2a6c1"}},{"code_sha256_prefix":"12febd0c96d3874c","entry":"kl_loss","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12febd0c96d3874c"}},{"code_sha256_prefix":"665b785d5b8b0b8f","entry":"zipf_loss","repo":"megvii-research/zipfls","repo_kind":"official","path":"pytorch_zipfls/losses.py","file_url":"https://github.com/megvii-research/zipfls/blob/HEAD/pytorch_zipfls/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"665b785d5b8b0b8f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}