{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-neural-network-training","title":"Benchmarking Neural Network Training Algorithms","arxiv_id":"2306.07179","date":"2023-06-12","proceeding":null,"authors":["George E. Dahl","Frank Schneider","Zachary Nado","Naman Agarwal","Chandramouli Shama Sastry","Philipp Hennig","Sourabh Medapati","Runa Eschenhagen","Priya Kasimbeg","Daniel Suo","Juhan Bae","Justin Gilmer","Abel L. Peirson","Bilal Khan","Rohan Anil","Mike Rabbat","Shankar Krishnan","Daniel Snider","Ehsan Amid","Kongtao Chen","Chris J. Maddison","Rakshith Vasudev","Michal Badura","Ankush Garg","Peter Mattson"],"abstract":"Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workloads (e.g., better update rules, tuning protocols, learning rate schedules, or data selection schemes) could save time, save computational resources, and lead to better, more accurate, models. Unfortunately, as a community, we are currently unable to reliably identify training algorithm improvements, or even determine the state-of-the-art training algorithm. In this work, using concrete experiments, we argue that real progress in speeding up training requires new benchmarks that resolve three basic challenges faced by empirical comparisons of training algorithms: (1) how to decide when training is complete and precisely measure training time, (2) how to handle the sensitivity of measurements to exact workload details, and (3) how to fairly compare algorithms that require hyperparameter tuning. In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of workload variants that make it possible to detect benchmark submissions that are more robust to workload changes than current widely-used methods. Finally, we evaluate baseline submissions constructed using various optimizers that represent current practice, as well as other optimizers that have recently received attention in the literature. These baseline results collectively demonstrate the feasibility of our benchmark, show that non-trivial gaps between methods exist, and set a provisional state-of-the-art for future benchmark submissions to try and surpass.","url_abs":"https://arxiv.org/abs/2306.07179v1","url_pdf":"https://arxiv.org/pdf/2306.07179v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-neural-network-training","repo_url":"https://github.com/mlcommons/algorithmic-efficiency","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"benchmarking-neural-network-training","repo_url":"https://github.com/deepmind/optax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"benchmarking-neural-network-training","repo_url":"https://github.com/google-deepmind/optax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"benchmarking-neural-network-training","repo_url":"https://github.com/mlcommons/submissions_algorithms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.07179","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.07179"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/optax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/submissions_algorithms","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/optax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/algorithmic-efficiency","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":22},"by_repo_kind":{"listed":{"samples":22,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8c1ff079e1c251cf","entry":"abs_sq","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/numerics.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/numerics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c1ff079e1c251cf"}},{"code_sha256_prefix":"52002c9d4813e1ec","entry":"adabelief","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/alias.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/alias.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52002c9d4813e1ec"}},{"code_sha256_prefix":"c1bf0f66f8ead4ab","entry":"adadelta","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/alias.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/alias.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1bf0f66f8ead4ab"}},{"code_sha256_prefix":"be540cbd764c7d2a","entry":"adafactor","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/alias.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/alias.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be540cbd764c7d2a"}},{"code_sha256_prefix":"46fa6da2a2e26515","entry":"deprecation_getattr","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/deprecations.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/deprecations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46fa6da2a2e26515"}},{"code_sha256_prefix":"34f71acccac437e4","entry":"global_norm","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linear_algebra.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linear_algebra.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34f71acccac437e4"}},{"code_sha256_prefix":"6cb62843159b4e5e","entry":"init_empty_state","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/base.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6cb62843159b4e5e"}},{"code_sha256_prefix":"f69cdcfdf348b5ba","entry":"lookahead","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/lookahead.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/lookahead.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f69cdcfdf348b5ba"}},{"code_sha256_prefix":"86e6aca17da03e77","entry":"matrix_inverse_pth_root","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linear_algebra.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linear_algebra.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86e6aca17da03e77"}},{"code_sha256_prefix":"ed4ddeade9eb2eee","entry":"power_iteration","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linear_algebra.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linear_algebra.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed4ddeade9eb2eee"}},{"code_sha256_prefix":"8446f6b71ea9fde1","entry":"safe_norm","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/numerics.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/numerics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8446f6b71ea9fde1"}},{"code_sha256_prefix":"b8071ff27a01c329","entry":"safe_root_mean_squares","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/numerics.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/numerics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8071ff27a01c329"}},{"code_sha256_prefix":"e41730968a3fffcb","entry":"scale_by_backtracking_linesearch","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linesearch.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linesearch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e41730968a3fffcb"}},{"code_sha256_prefix":"32fce6c5f1050dd1","entry":"scale_by_factored_rms","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/factorized.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/factorized.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32fce6c5f1050dd1"}},{"code_sha256_prefix":"0c22d7c2cc2a90d0","entry":"scale_by_rms","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/transform.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c22d7c2cc2a90d0"}},{"code_sha256_prefix":"49a32735e1eb328a","entry":"scale_by_rss","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/transform.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49a32735e1eb328a"}},{"code_sha256_prefix":"670803584425b73a","entry":"scale_by_stddev","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/transform.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"670803584425b73a"}},{"code_sha256_prefix":"3fcd57975548149b","entry":"scale_by_zoom_linesearch","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linesearch.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linesearch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fcd57975548149b"}},{"code_sha256_prefix":"08a168538fed09eb","entry":"stateless","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/base.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08a168538fed09eb"}},{"code_sha256_prefix":"8dcf52b4ebe33a4a","entry":"stateless_with_tree_map","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/base.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8dcf52b4ebe33a4a"}},{"code_sha256_prefix":"db7af5450cd8ac59","entry":"warn_deprecated_function","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/deprecations.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/deprecations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db7af5450cd8ac59"}},{"code_sha256_prefix":"5760b69d788a57d7","entry":"zoom_linesearch","repo":"deepmind/optax","repo_kind":"listed","path":"optax/_src/linesearch.py","file_url":"https://github.com/deepmind/optax/blob/HEAD/optax/_src/linesearch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5760b69d788a57d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}