{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zeroth-order-optimization-finds-flat-minima","title":"Zeroth-Order Optimization Finds Flat Minima","arxiv_id":"2506.05454","date":"2025-06-05","proceeding":null,"authors":["Liang Zhang","Bingcong Li","Kiran Koshy Thekumparampil","Sewoong Oh","Michael Muehlebach","Niao He"],"abstract":"Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization theory focuses on convergence to an arbitrary stationary point, but less is known on the implicit regularization that provides a fine-grained characterization on which particular solutions are finally reached. We show that zeroth-order optimization with the standard two-point estimator favors solutions with small trace of Hessian, which is widely used in previous work to distinguish between sharp and flat minima. We further provide convergence rates of zeroth-order optimization to approximate flat minima for convex and sufficiently smooth functions, where flat minima are defined as the minimizers that achieve the smallest trace of Hessian among all optimal solutions. Experiments on binary classification tasks with convex losses and language model fine-tuning support our theoretical findings.","url_abs":"https://arxiv.org/abs/2506.05454v1","url_pdf":"https://arxiv.org/pdf/2506.05454v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.05454","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.05454"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Liang137/FlatZero","reach":null}],"summary":{"ran_violates":3,"ran_fixture":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a373a6ed03c9cc41","entry":"compute_grad","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"svm_logreg/logreg.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/svm_logreg/logreg.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a373a6ed03c9cc41"}},{"code_sha256_prefix":"3ebcc930731af124","entry":"compute_loss","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"svm_logreg/logreg.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/svm_logreg/logreg.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ebcc930731af124"}},{"code_sha256_prefix":"68cef691420d81ee","entry":"loss","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"test_func/run_toy.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/test_func/run_toy.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68cef691420d81ee"}},{"code_sha256_prefix":"a911f9cacbc9faf4","entry":"sigmoid","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"svm_logreg/logreg.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/svm_logreg/logreg.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a911f9cacbc9faf4"}},{"code_sha256_prefix":"f05d08ea970e9e2f","entry":"grad","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"test_func/run_toy.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/test_func/run_toy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f05d08ea970e9e2f"}},{"code_sha256_prefix":"bb28c94793eae7dd","entry":"zo_grad","repo":"Liang137/FlatZero","repo_kind":"found_in_text","path":"test_func/run_toy.py","file_url":"https://github.com/Liang137/FlatZero/blob/HEAD/test_func/run_toy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb28c94793eae7dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}