{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parameter-re-initialization-through-cyclical","title":"Parameter Re-Initialization through Cyclical Batch Size Schedules","arxiv_id":"1812.01216","date":"2018-12-04","proceeding":null,"authors":["Norman Mu","Zhewei Yao","Amir Gholami","Kurt Keutzer","Michael Mahoney"],"abstract":"Optimal parameter initialization remains a crucial problem for neural network\ntraining. A poor weight initialization may take longer to train and/or converge\nto sub-optimal solutions. Here, we propose a method of weight re-initialization\nby repeated annealing and injection of noise in the training process. We\nimplement this through a cyclical batch size schedule motivated by a Bayesian\nperspective of neural network training. We evaluate our methods through\nextensive experiments on tasks in language modeling, natural language\ninference, and image classification. We demonstrate the ability of our method\nto improve language modeling performance by up to 7.91 perplexity and reduce\ntraining iterations by up to $61\\%$, in addition to its flexibility in enabling\nsnapshot ensembling and use with adversarial training.","url_abs":"http://arxiv.org/abs/1812.01216v1","url_pdf":"http://arxiv.org/pdf/1812.01216v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-inference-on-snli","task":"Natural Language Inference","dataset":"SNLI","model":"CBS-1 + ESIM","rank_in_archive_order":51,"of":98,"metrics":{"% Test Accuracy":"86.73"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.01216","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}