{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalization-matters-loss-minima-flattening","title":"Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation","arxiv_id":"2303.14666","date":"2023-03-26","proceeding":"CVPR 2023 1","authors":["Tianli Zhang","Mengqi Xue","Jiangtao Zhang","Haofei Zhang","Yu Wang","Lechao Cheng","Jie Song","Mingli Song"],"abstract":"Most existing online knowledge distillation(OKD) techniques typically require sophisticated modules to produce diverse knowledge for improving students' generalization ability. In this paper, we strive to fully utilize multi-model settings instead of well-designed modules to achieve a distillation effect with excellent generalization performance. Generally, model generalization can be reflected in the flatness of the loss landscape. Since averaging parameters of multiple models can find flatter minima, we are inspired to extend the process to the sampled convex combinations of multi-student models in OKD. Specifically, by linearly weighting students' parameters in each training batch, we construct a Hybrid-Weight Model(HWM) to represent the parameters surrounding involved students. The supervision loss of HWM can estimate the landscape's curvature of the whole region around students to measure the generalization explicitly. Hence we integrate HWM's loss into students' training and propose a novel OKD framework via parameter hybridization(OKDPH) to promote flatter minima and obtain robust solutions. Considering the redundancy of parameters could lead to the collapse of HWM, we further introduce a fusion operation to keep the high similarity of students. Compared to the state-of-the-art(SOTA) OKD methods and SOTA methods of seeking flat minima, our OKDPH achieves higher performance with fewer parameters, benefiting OKD with lightweight and robust characteristics. Our code is publicly available at https://github.com/tianlizhang/OKDPH.","url_abs":"https://arxiv.org/abs/2303.14666v1","url_pdf":"https://arxiv.org/pdf/2303.14666v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generalization-matters-loss-minima-flattening","repo_url":"https://github.com/tianlizhang/okdph","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.14666","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.14666"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tianlizhang/OKDPH","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"named_in_paper":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1247dbd0653b5938","entry":"fusion","repo":"tianlizhang/OKDPH","repo_kind":"named_in_paper","path":"src/OKDPH.py","file_url":"https://github.com/tianlizhang/OKDPH/blob/HEAD/src/OKDPH.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1247dbd0653b5938"}},{"code_sha256_prefix":"0958a4ac84d6a974","entry":"train","repo":"tianlizhang/OKDPH","repo_kind":"named_in_paper","path":"src/OKDPH.py","file_url":"https://github.com/tianlizhang/OKDPH/blob/HEAD/src/OKDPH.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0958a4ac84d6a974"}},{"code_sha256_prefix":"69998480f1f92ddf","entry":"update_hwm","repo":"tianlizhang/OKDPH","repo_kind":"named_in_paper","path":"src/OKDPH.py","file_url":"https://github.com/tianlizhang/OKDPH/blob/HEAD/src/OKDPH.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69998480f1f92ddf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}