{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-generalization-and-convergence-by","title":"Improving Generalization and Convergence by Enhancing Implicit Regularization","arxiv_id":"2405.20763","date":"2024-05-31","proceeding":null,"authors":["Mingze Wang","Jinbo Wang","Haotian He","Zilin Wang","Guanhua Huang","Feiyu Xiong","Zhiyu Li","Weinan E","Lei Wu"],"abstract":"In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with {\\em generic base optimizers} without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a $2\\times$ {\\em speed-up} compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).","url_abs":"https://arxiv.org/abs/2405.20763v4","url_pdf":"https://arxiv.org/pdf/2405.20763v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-generalization-and-convergence-by","repo_url":"https://github.com/wmz9/ire-algorithm-framework","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"sharpness-aware-minimization","method_name":"Sharpness-Aware Minimization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wmz9/IRE-algorithm-framework","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wmz9/ire-algorithm-framework","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":2,"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"wmz9/IRE-algorithm-framework","repo_kind":"official","path":"NLP/LLM/modelling_llama_new.py","file_url":"https://github.com/wmz9/IRE-algorithm-framework/blob/HEAD/NLP/LLM/modelling_llama_new.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"ddcb8bf702ecd1b2","entry":"IRE_PARAMS","repo":"wmz9/ire-algorithm-framework","repo_kind":"official","path":"CV/imagenet/ire.py","file_url":"https://github.com/wmz9/ire-algorithm-framework/blob/HEAD/CV/imagenet/ire.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddcb8bf702ecd1b2"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"wmz9/IRE-algorithm-framework","repo_kind":"official","path":"NLP/LLM/modelling_llama_new.py","file_url":"https://github.com/wmz9/IRE-algorithm-framework/blob/HEAD/NLP/LLM/modelling_llama_new.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"5ae6fc518f0d2f24","entry":"get_lr","repo":"wmz9/ire-algorithm-framework","repo_kind":"official","path":"NLP/LLM/train_admire_wiki103_Llama.py","file_url":"https://github.com/wmz9/ire-algorithm-framework/blob/HEAD/NLP/LLM/train_admire_wiki103_Llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ae6fc518f0d2f24"}},{"code_sha256_prefix":"74852eac140a708f","entry":"get_rotary_matrix","repo":"wmz9/IRE-algorithm-framework","repo_kind":"official","path":"NLP/LLM/modelling_Llama_hand.py","file_url":"https://github.com/wmz9/IRE-algorithm-framework/blob/HEAD/NLP/LLM/modelling_Llama_hand.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74852eac140a708f"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"wmz9/IRE-algorithm-framework","repo_kind":"official","path":"NLP/LLM/modelling_llama_new.py","file_url":"https://github.com/wmz9/IRE-algorithm-framework/blob/HEAD/NLP/LLM/modelling_llama_new.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"f4dd60141fdfbf0b","entry":"smooth_crossentropy","repo":"wmz9/IRE-algorithm-framework","repo_kind":"official","path":"CV/cifar/model/smooth_cross_entropy.py","file_url":"https://github.com/wmz9/IRE-algorithm-framework/blob/HEAD/CV/cifar/model/smooth_cross_entropy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f4dd60141fdfbf0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}