{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neuroevobench-benchmarking-evolutionary","title":"NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications","arxiv_id":"2311.02394","date":"2023-11-04","proceeding":"NeurIPS 2023 11","authors":["Robert Tjarko Lange","Yujin Tang","Yingtao Tian"],"abstract":"Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differentiable operators. One core reason for this trend has been the recent innovation in hardware acceleration and compatible software - making distributed population evaluations much easier than before. Unlike for gradient descent-based methods though, there is a lack of hyperparameter understanding and best practices for EO - arguably due to severely less 'graduate student descent' and benchmarking being performed for EO methods. Additionally, classical benchmarks from the evolutionary community provide few practical insights for Deep Learning applications. This poses challenges for newcomers to hardware-accelerated EO and hinders significant adoption. Hence, we establish a new benchmark of EO methods (NeuroEvoBench) tailored toward Deep Learning applications and exhaustively evaluate traditional and meta-learned EO. We investigate core scientific questions including resource allocation, fitness shaping, normalization, regularization & scalability of EO. The benchmark is open-sourced at https://github.com/neuroevobench/neuroevobench under Apache-2.0 license.","url_abs":"https://arxiv.org/abs/2311.02394v1","url_pdf":"https://arxiv.org/pdf/2311.02394v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neuroevobench-benchmarking-evolutionary","repo_url":"https://github.com/neuroevobench/neuroevobench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.02394","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.02394"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/neuroevobench/neuroevobench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff29396687b2365e","entry":"exp_quadratic","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/blines/bo_utils/bo_gp.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/blines/bo_utils/bo_gp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff29396687b2365e"}},{"code_sha256_prefix":"105d8c9c5f766d1b","entry":"plot_task","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/postprocess/plot.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/postprocess/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"105d8c9c5f766d1b"}},{"code_sha256_prefix":"a02278fd73f185a5","entry":"softplus","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/blines/bo_utils/bo_gp.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/blines/bo_utils/bo_gp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a02278fd73f185a5"}},{"code_sha256_prefix":"6837e7a013f4566d","entry":"yaml_log_to_pd","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/postprocess/filter.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/postprocess/filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6837e7a013f4566d"}},{"code_sha256_prefix":"d3f372bc7783c73e","entry":"filter_best","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/postprocess/filter.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/postprocess/filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3f372bc7783c73e"}},{"code_sha256_prefix":"81e9c26876bcefac","entry":"get_search_history","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/postprocess/filter.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/postprocess/filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81e9c26876bcefac"}},{"code_sha256_prefix":"e1f61d5c5ba61af4","entry":"plot_interval_estimates","repo":"neuroevobench/neuroevobench","repo_kind":"official","path":"neuroevobench/metrics/rliab.py","file_url":"https://github.com/neuroevobench/neuroevobench/blob/HEAD/neuroevobench/metrics/rliab.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1f61d5c5ba61af4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}