{"url":"/method/t-fixup","slug":"t-fixup","name":"T-Fixup","full_name":"T-Fixup","full_name_withheld":false,"description_markdown":"**T-Fixup** is an [initialization](https://paperswithcode.com/methods/category/initialization) method for [Transformers](https://paperswithcode.com/methods/category/transformers) that aims to remove the need for [layer normalization](https://paperswithcode.com/method/layer-normalization) and [warmup](https://paperswithcode.com/method/linear-warmup). The initialization procedure is as follows:\r\n\r\n- Apply [Xavier initialization](https://paperswithcode.com/method/xavier-initialization) for all parameters excluding input embeddings. Use Gaussian initialization $\\mathcal{N}\\left(0, d^{-\\frac{1}{2}}\\right)$ for input embeddings where $d$ is the embedding dimension.\r\n- Scale $\\mathbf{v}\\_{d}$ and $\\mathbf{w}\\_{d}$ matrices in each decoder [attention block](https://paperswithcode.com/method/multi-head-attention), weight matrices in each decoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) and input embeddings $\\mathbf{x}$ and $\\mathbf{y}$ in encoder and decoder by $(9 N)^{-\\frac{1}{4}}$\r\n- Scale $\\mathbf{v}\\_{e}$ and $\\mathbf{w}\\_{e}$ matrices in each encoder [attention block](https://paperswithcode.com/method/multi-head-attention) and weight matrices in each encoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) by $0.67 N^{-\\frac{1}{4}}$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Improving Transformer Optimization Through Better Initialization","paper":"/paper/improving-transformer-optimization-through-1","first_author":"Xiao Shi Huang","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/improving-transformer-optimization-through-1"},"source":{"url":"https://proceedings.icml.cc/static/paper_files/icml/2020/5691-Paper.pdf","title":"Improving Transformer Optimization Through Better Initialization","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Initialization","url":"/methods/category/initialization","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":"/paper/optimizing-deeper-transformers-on-small","title":"Optimizing Deeper Transformers on Small Datasets","date":"2020-12-30","arxiv_id":"2012.15355","n_code_links":1,"syntology":null},{"paper":"/paper/improving-transformer-optimization-through-1","title":"Improving Transformer Optimization Through Better Initialization","date":"2020-01-01","arxiv_id":null,"n_code_links":1,"syntology":null}],"papers_shown":2,"tasks":[{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/reading-comprehension","name":"Reading Comprehension","papers":1},{"task":"/task/sql-parsing","name":"SQL Parsing","papers":1},{"task":"/task/semantic-parsing","name":"Semantic Parsing","papers":1},{"task":"/task/text-to-sql","name":"Text to SQL","papers":1},{"task":"/task/text-to-sql","name":"Text-To-SQL","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2020","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/t-fixup"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}