{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mesh-tensorflow-deep-learning-for","title":"Mesh-TensorFlow: Deep Learning for Supercomputers","arxiv_id":"1811.02084","date":"2018-11-05","proceeding":"NeurIPS 2018 12","authors":["Noam Shazeer","Youlong Cheng","Niki Parmar","Dustin Tran","Ashish Vaswani","Penporn Koanantakool","Peter Hawkins","HyoukJoong Lee","Mingsheng Hong","Cliff Young","Ryan Sepassi","Blake Hechtman"],"abstract":"Batch-splitting (data-parallelism) is the dominant distributed Deep Neural\nNetwork (DNN) training strategy, due to its universal applicability and its\namenability to Single-Program-Multiple-Data (SPMD) programming. However,\nbatch-splitting suffers from problems including the inability to train very\nlarge models (due to memory constraints), high latency, and inefficiency at\nsmall batch sizes. All of these can be solved by more general distribution\nstrategies (model-parallelism). Unfortunately, efficient model-parallel\nalgorithms tend to be complicated to discover, describe, and to implement,\nparticularly on large clusters. We introduce Mesh-TensorFlow, a language for\nspecifying a general class of distributed tensor computations. Where\ndata-parallelism can be viewed as splitting tensors and operations along the\n\"batch\" dimension, in Mesh-TensorFlow, the user can specify any\ntensor-dimensions to be split across any dimensions of a multi-dimensional mesh\nof processors. A Mesh-TensorFlow graph compiles into a SPMD program consisting\nof parallel operations coupled with collective communication primitives such as\nAllreduce. We use Mesh-TensorFlow to implement an efficient data-parallel,\nmodel-parallel version of the Transformer sequence-to-sequence model. Using TPU\nmeshes of up to 512 cores, we train Transformer models with up to 5 billion\nparameters, surpassing state of the art results on WMT'14 English-to-French\ntranslation task and the one-billion-word language modeling benchmark.\nMesh-Tensorflow is available at https://github.com/tensorflow/mesh .","url_abs":"http://arxiv.org/abs/1811.02084v1","url_pdf":"http://arxiv.org/pdf/1811.02084v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mesh-tensorflow-deep-learning-for","repo_url":"https://github.com/tensorflow/mesh","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mesh-tensorflow","method_name":"Mesh-TensorFlow"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"mesh-tensorflow","name":"Mesh-TensorFlow","full_name":"Mesh-TensorFlow"}],"results":[{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"Mesh Tensorflow","rank_in_archive_order":12,"of":27,"metrics":{"Number of params":"4.9B","PPL":" 24.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1811.02084","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}