{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-tips-for-the-transformer-model","title":"Training Tips for the Transformer Model","arxiv_id":"1804.00247","date":"2018-04-01","proceeding":null,"authors":["Martin Popel","Ondřej Bojar"],"abstract":"This article describes our experiments in neural machine translation using\nthe recent Tensor2Tensor framework and the Transformer sequence-to-sequence\nmodel (Vaswani et al., 2017). We examine some of the critical parameters that\naffect the final translation quality, memory usage, training stability and\ntraining time, concluding each experiment with a set of recommendations for\nfellow researchers. In addition to confirming the general mantra \"more data and\nlarger models\", we address scaling to multiple GPUs and provide practical tips\nfor improved training regarding batch size, learning rate, warmup steps,\nmaximum sentence length and checkpoint averaging. We hope that our observations\nwill allow others to get better results given their particular hardware and\ndata constraints.","url_abs":"http://arxiv.org/abs/1804.00247v2","url_pdf":"http://arxiv.org/pdf/1804.00247v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-tips-for-the-transformer-model","repo_url":"https://github.com/tensorflow/tensor2tensor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"training-tips-for-the-transformer-model","repo_url":"https://github.com/awsm-research/autotransform-replication","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"training-tips-for-the-transformer-model","repo_url":"https://github.com/caoyujiALgLM/NLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"training-tips-for-the-transformer-model","repo_url":"https://github.com/gavincaoyuji/NLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.00247","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}