{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/questions-for-flat-minima-optimization-of","title":"When Do Flat Minima Optimizers Work?","arxiv_id":"2202.00661","date":"2022-02-01","proceeding":null,"authors":["Jean Kaddour","Linqing Liu","Ricardo Silva","Matt J. Kusner"],"abstract":"Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.","url_abs":"https://arxiv.org/abs/2202.00661v5","url_pdf":"https://arxiv.org/pdf/2202.00661v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"questions-for-flat-minima-optimization-of","repo_url":"https://github.com/JeanKaddour/WASAM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"graph-learning","task_name":"Graph Learning"},{"task_slug":"graph-representation-learning","task_name":"Graph Representation Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"node-classification","task_name":"Node Classification"},{"task_slug":"node-property-prediction","task_name":"Node Property Prediction"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"}],"methods":[{"method_slug":"sharpness-aware-minimization","method_name":"Sharpness-Aware Minimization"},{"method_slug":"stochastic-weight-averaging","method_name":"Stochastic Weight Averaging"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.00661","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}