{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-the-difficulty-of-training","title":"Understanding the Difficulty of Training Transformers","arxiv_id":"2004.08249","date":"2020-04-17","proceeding":"EMNLP 2020 11","authors":["Liyuan Liu","Xiaodong Liu","Jianfeng Gao","Weizhu Chen","Jiawei Han"],"abstract":"Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers carefully (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand $\\textit{what complicates Transformer training}$ from both empirical and theoretical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that influences training substantially -- for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output. Yet we observe that a light dependency limits the model potential and leads to inferior trained models. Inspired by our analysis, we propose Admin ($\\textbf{Ad}$aptive $\\textbf{m}$odel $\\textbf{in}$itialization) to stabilize stabilize the early stage's training and unleash its full potential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance. Implementations are released at: https://github.com/LiyuanLucasLiu/Transforemr-Clinic.","url_abs":"https://arxiv.org/abs/2004.08249v3","url_pdf":"https://arxiv.org/pdf/2004.08249v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-the-difficulty-of-training","repo_url":"https://github.com/LiyuanLucasLiu/Transforemr-Clinic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"understanding-the-difficulty-of-training","repo_url":"https://github.com/c00k1ez/plain-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"Admin","rank_in_archive_order":6,"of":57,"metrics":{"BLEU score":"43.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.08249","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.08249"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LiyuanLucasLiu/Transforemr-Clinic","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/c00k1ez/plain-transformers","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2378be1d254fb13c","entry":"FFN","repo":"c00k1ez/plain-transformers","repo_kind":"listed","path":"src/plain_transformers/layers/pre_ln_encoder.py","file_url":"https://github.com/c00k1ez/plain-transformers/blob/HEAD/src/plain_transformers/layers/pre_ln_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2378be1d254fb13c"}},{"code_sha256_prefix":"d3e534b847d15575","entry":"MultiHeadAttention","repo":"c00k1ez/plain-transformers","repo_kind":"listed","path":"src/plain_transformers/layers/pre_ln_encoder.py","file_url":"https://github.com/c00k1ez/plain-transformers/blob/HEAD/src/plain_transformers/layers/pre_ln_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3e534b847d15575"}},{"code_sha256_prefix":"b8547574c067ca6c","entry":"RAdam","repo":"LiyuanLucasLiu/Transforemr-Clinic","repo_kind":"official","path":"radam_fairseq/radam.py","file_url":"https://github.com/LiyuanLucasLiu/Transforemr-Clinic/blob/HEAD/radam_fairseq/radam.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8547574c067ca6c"}},{"code_sha256_prefix":"fc0804364f10665a","entry":"act_to_func","repo":"c00k1ez/plain-transformers","repo_kind":"listed","path":"src/plain_transformers/layers/pre_ln_encoder.py","file_url":"https://github.com/c00k1ez/plain-transformers/blob/HEAD/src/plain_transformers/layers/pre_ln_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc0804364f10665a"}},{"code_sha256_prefix":"5f94447ee64fb0d9","entry":"PreLNEncoderLayer","repo":"c00k1ez/plain-transformers","repo_kind":"listed","path":"src/plain_transformers/layers/pre_ln_encoder.py","file_url":"https://github.com/c00k1ez/plain-transformers/blob/HEAD/src/plain_transformers/layers/pre_ln_encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f94447ee64fb0d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}