{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deepzero-scaling-up-zeroth-order-optimization","title":"DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training","arxiv_id":"2310.02025","date":"2023-10-03","proceeding":null,"authors":["Aochuan Chen","Yimeng Zhang","Jinghan Jia","James Diffenderfer","Jiancheng Liu","Konstantinos Parasyris","Yihua Zhang","Zheng Zhang","Bhavya Kailkhura","Sijia Liu"],"abstract":"Zeroth-order (ZO) optimization has become a popular technique for solving machine learning (ML) problems when first-order (FO) information is difficult or impossible to obtain. However, the scalability of ZO optimization remains an open problem: Its use has primarily been limited to relatively small-scale ML problems, such as sample-wise adversarial attack generation. To our best knowledge, no prior work has demonstrated the effectiveness of ZO optimization in training deep neural networks (DNNs) without a significant decrease in performance. To overcome this roadblock, we develop DeepZero, a principled ZO deep learning (DL) framework that can scale ZO optimization to DNN training from scratch through three primary innovations. First, we demonstrate the advantages of coordinatewise gradient estimation (CGE) over randomized vector-wise gradient estimation in training accuracy and computational efficiency. Second, we propose a sparsityinduced ZO training protocol that extends the model pruning methodology using only finite differences to explore and exploit the sparse DL prior in CGE. Third, we develop the methods of feature reuse and forward parallelization to advance the practical implementations of ZO training. Our extensive experiments show that DeepZero achieves state-of-the-art (SOTA) accuracy on ResNet-20 trained on CIFAR-10, approaching FO training performance for the first time. Furthermore, we show the practical utility of DeepZero in applications of certified adversarial defense and DL-based partial differential equation error correction, achieving 10-20% improvement over SOTA. We believe our results will inspire future research on scalable ZO optimization and contribute to advancing DL with black box. Codes are available at https://github.com/OPTML-Group/DeepZero.","url_abs":"https://arxiv.org/abs/2310.02025v4","url_pdf":"https://arxiv.org/pdf/2310.02025v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deepzero-scaling-up-zeroth-order-optimization","repo_url":"https://github.com/OPTML-Group/DeepZero","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-defense","task_name":"Adversarial Defense"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"physical-simulations","task_name":"Physical Simulations"}],"methods":[{"method_slug":"deepzero","method_name":"DeepZero"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[{"slug":"deepzero","name":"DeepZero","full_name":"DeepZero"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02025","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02025"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tum-pbs/Solver-in-the-Loop","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OPTML-Group/DeepZero","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"813dc6ef3f8cc022","entry":"cge","repo":"OPTML-Group/DeepZero","repo_kind":"official","path":"algorithm/zoo/gradient_estimate.py","file_url":"https://github.com/OPTML-Group/DeepZero/blob/HEAD/algorithm/zoo/gradient_estimate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"813dc6ef3f8cc022"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}