{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-efficient-and-scalable-sharpness","title":"Towards Efficient and Scalable Sharpness-Aware Minimization","arxiv_id":"2203.02714","date":"2022-03-05","proceeding":"CVPR 2022 1","authors":["Yong liu","Siqi Mai","Xiangning Chen","Cho-Jui Hsieh","Yang You"],"abstract":"Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that LookSAM achieves similar accuracy gains to SAM while being tremendously faster - it enjoys comparable computational complexity with first-order optimizers such as SGD or Adam. To further evaluate the performance and scalability of LookSAM, we incorporate a layer-wise modification and perform experiments in the large-batch training scenario, which is more prone to converge to sharp local minima. We are the first to successfully scale up the batch size when training Vision Transformers (ViTs). With a 64k batch size, we are able to train ViTs from scratch in minutes while maintaining competitive performance.","url_abs":"https://arxiv.org/abs/2203.02714v1","url_pdf":"https://arxiv.org/pdf/2203.02714v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-efficient-and-scalable-sharpness","repo_url":"https://github.com/Leminhbinh0209/LookSAM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"towards-efficient-and-scalable-sharpness","repo_url":"https://github.com/rollovd/LookSAM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"towards-efficient-and-scalable-sharpness","repo_url":"https://github.com/MindSpore-scientific/code-14/tree/main/Scalable-Sharpness-Aware-Minimization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"towards-efficient-and-scalable-sharpness","repo_url":"https://github.com/MindSpore-scientific/code-14/tree/main/SeesawLoss-master","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"sharpness-aware-minimization","method_name":"Sharpness-Aware Minimization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.02714","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.02714"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rollovd/LookSAM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-14/tree/main/SeesawLoss-master","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-14/tree/main/Scalable-Sharpness-Aware-Minimization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Leminhbinh0209/LookSAM","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yong-6/LookSAM","reach":{"status":"ok"}}],"summary":{"ran":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"50385563753b0220","entry":"LookSAM","repo":"Leminhbinh0209/LookSAM","repo_kind":"listed","path":"looksam.py","file_url":"https://github.com/Leminhbinh0209/LookSAM/blob/HEAD/looksam.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50385563753b0220"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}