{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mlperf-training-benchmark","title":"MLPerf Training Benchmark","arxiv_id":"1910.01500","date":"2019-10-02","proceeding":null,"authors":["Peter Mattson","Christine Cheng","Cody Coleman","Greg Diamos","Paulius Micikevicius","David Patterson","Hanlin Tang","Gu-Yeon Wei","Peter Bailis","Victor Bittorf","David Brooks","Dehao Chen","Debojyoti Dutta","Udit Gupta","Kim Hazelwood","Andrew Hock","Xinyuan Huang","Atsushi Ike","Bill Jia","Daniel Kang","David Kanter","Naveen Kumar","Jeffery Liao","Guokai Ma","Deepak Narayanan","Tayo Oguntebi","Gennady Pekhimenko","Lillian Pentecost","Vijay Janapa Reddi","Taylor Robie","Tom St. John","Tsuguchika Tabaru","Carole-Jean Wu","Lingjie Xu","Masafumi Yamazaki","Cliff Young","Matei Zaharia"],"abstract":"Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.","url_abs":"https://arxiv.org/abs/1910.01500v3","url_pdf":"https://arxiv.org/pdf/1910.01500v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mlperf-training-benchmark","repo_url":"https://github.com/mlperf/training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mlperf-training-benchmark","repo_url":"https://github.com/mlcommons/training","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.01500","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.01500"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/training","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlperf/training","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"43b5c25ae15d4649","entry":"parse_envvars","repo":"mlcommons/training","repo_kind":"listed","path":"llm_moe_pretraining/nemo/run_deepseek.py","file_url":"https://github.com/mlcommons/training/blob/HEAD/llm_moe_pretraining/nemo/run_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43b5c25ae15d4649"}},{"code_sha256_prefix":"a5639ef83a9f70a4","entry":"parse_mounts","repo":"mlcommons/training","repo_kind":"listed","path":"llm_moe_pretraining/nemo/run_deepseek.py","file_url":"https://github.com/mlcommons/training/blob/HEAD/llm_moe_pretraining/nemo/run_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5639ef83a9f70a4"}},{"code_sha256_prefix":"8b82aa9163296277","entry":"create_loop_fn","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b82aa9163296277"}},{"code_sha256_prefix":"97fbdcb7caade0b9","entry":"create_tf_while_loop_fn","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97fbdcb7caade0b9"}},{"code_sha256_prefix":"9deb7b7f6b8bd358","entry":"fmt_size","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/mixtral8x22b/model_utils_tpu.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/mixtral8x22b/model_utils_tpu.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9deb7b7f6b8bd358"}},{"code_sha256_prefix":"51d3bc0cfa6415a3","entry":"gelu","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/bert/modeling.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51d3bc0cfa6415a3"}},{"code_sha256_prefix":"217fe96a5945a42d","entry":"get_activation","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/bert/modeling.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"217fe96a5945a42d"}},{"code_sha256_prefix":"7bc46f4950599a87","entry":"get_assignment_map_from_checkpoint","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/bert/modeling.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bc46f4950599a87"}},{"code_sha256_prefix":"29c2544a914634df","entry":"get_optimizer","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/unet3d/pytorch/runtime/training.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/unet3d/pytorch/runtime/training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29c2544a914634df"}},{"code_sha256_prefix":"db8154c629e73aca","entry":"make_distributed_dataset","repo":"mlperf/training","repo_kind":"official","path":"retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","file_url":"https://github.com/mlperf/training/blob/HEAD/retired_benchmarks/resnet-tf2/tensorflow2/tf2_common/training/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db8154c629e73aca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}