{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-non-parametric-optimizer-search-for","title":"Efficient Non-Parametric Optimizer Search for Diverse Tasks","arxiv_id":"2209.13575","date":"2022-09-27","proceeding":null,"authors":["Ruochen Wang","Yuanhao Xiong","Minhao Cheng","Cho-Jui Hsieh"],"abstract":"Efficient and automated design of optimizers plays a crucial role in full-stack AutoML systems. However, prior methods in optimizer search are often limited by their scalability, generability, or sample efficiency. With the goal of democratizing research and application of optimizer search, we present the first efficient, scalable and generalizable framework that can directly search on the tasks of interest. We first observe that optimizer updates are fundamentally mathematical expressions applied to the gradient. Inspired by the innate tree structure of the underlying math expressions, we re-arrange the space of optimizers into a super-tree, where each path encodes an optimizer. This way, optimizer search can be naturally formulated as a path-finding problem, allowing a variety of well-established tree traversal methods to be used as the search algorithm. We adopt an adaptation of the Monte Carlo method to tree search, equipped with rejection sampling and equivalent- form detection that leverage the characteristics of optimizer update rules to further boost the sample efficiency. We provide a diverse set of tasks to benchmark our algorithm and demonstrate that, with only 128 evaluations, the proposed framework can discover optimizers that surpass both human-designed counterparts and prior optimizer search methods.","url_abs":"https://arxiv.org/abs/2209.13575v1","url_pdf":"https://arxiv.org/pdf/2209.13575v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-non-parametric-optimizer-search-for","repo_url":"https://github.com/ruocwang/efficient-optimizer-search","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"automl","task_name":"AutoML"},{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.13575","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.13575"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruocwang/efficient-optimizer-search","reach":null}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4465536a110c6ef4","entry":"ProgramLearningAlgorithm","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4465536a110c6ef4"}},{"code_sha256_prefix":"8191438f2eae0781","entry":"ProgramNode","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8191438f2eae0781"}},{"code_sha256_prefix":"a07a253ee8057616","entry":"Visit","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a07a253ee8057616"}},{"code_sha256_prefix":"5f8b23db306d8206","entry":"MC_SAMPLING_NOS","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f8b23db306d8206"}},{"code_sha256_prefix":"c5929cfc13a96de4","entry":"ProgramGraph","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5929cfc13a96de4"}},{"code_sha256_prefix":"a73f58271e56d4a3","entry":"load_training_task","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a73f58271e56d4a3"}},{"code_sha256_prefix":"ad6c9f7884554a1f","entry":"log_and_print","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad6c9f7884554a1f"}},{"code_sha256_prefix":"90e91c9cf039b628","entry":"print_program","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90e91c9cf039b628"}},{"code_sha256_prefix":"845bc5b5aadd180c","entry":"print_program_dict","repo":"ruocwang/efficient-optimizer-search","repo_kind":"official","path":"src/algorithms/nos_mc_sampling.py","file_url":"https://github.com/ruocwang/efficient-optimizer-search/blob/HEAD/src/algorithms/nos_mc_sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"845bc5b5aadd180c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}