Methods › General › Parameter Sharing › Weight Tying
Weight Tying
Introduced by Ofir Press et al. in Using the Output Embedding to Improve Language Models
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Weight Tying improves the performance of language models by tying (sharing) the weights of the embedding and softmax layers. This method also massively reduces the total number of parameters in the language models that it is applied to.
Language models are typically comprised of an embedding layer, followed by a number of Transformer or LSTM layers, which are finally followed by a softmax layer. Embedding layers learn word representations, such that similar words (in meaning) are represented by vectors that are near each other (in cosine distance). [Press & Wolf, 2016] showed that the softmax matrix, in which every word also has a vector representation, also exhibits this property. This leads them to propose to share the softmax and embedding matrices, which is done today in nearly all language models.
This method was independently introduced by Press & Wolf, 2016 and Inan et al, 2016.
Additionally, the Press & Wolf paper proposes Three-way Weight Tying, a method for NMT models in which the embedding matrix for the source language, the embedding matrix for the target language, and the softmax matrix for the target language are all tied. That method has been adopted by the Attention Is All You Need model and many other neural machine translation models.
Papers archive 2025-07-28
30 shown of 63, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Advanced Deep Learning Techniques for Analyzing Earnings Call Transcripts: Methodologies and Applications 27 Feb 2025 · 0 repositories · arXiv:2503.01886
-
No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts 24 Oct 2024 · 0 repositories · arXiv:2410.19184
-
RICo: Reddit ideological communities 5 Jun 2024 · 1 repository
-
Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study 15 Dec 2023 · 0 repositories
-
Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information 8 Dec 2023 · 0 repositories · arXiv:2312.04944
-
Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying 16 Nov 2023 · 0 repositories · arXiv:2311.09578
-
Headless Language Models: Learning without Predicting with Contrastive Weight Tying 15 Sep 2023 · 0 repositories · arXiv:2309.08351
-
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation 30 Jun 2023 · 2 repositories · arXiv:2306.17817
-
Explainable and High-Performance Hate and Offensive Speech Detection 26 Jun 2022 · 0 repositories · arXiv:2206.12983
-
Approximately Equivariant Networks for Imperfectly Symmetric Dynamics 28 Jan 2022 · 1 repository · arXiv:2201.11969Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
-
IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages 15 Nov 2021 · 1 repository · arXiv:2111.07906
-
Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling 27 Aug 2021 · 1 repository · arXiv:2108.12177
-
Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts 24 Aug 2021 · 1 repository · arXiv:2108.10939
-
Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System 20 Jul 2021 · 0 repositories · arXiv:2107.09625
-
WHOSe Heritage: Classification of UNESCO World Heritage "Outstanding Universal Value" Documents with Soft Labels 12 Apr 2021 · 1 repository · arXiv:2104.05547
-
L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset 21 Mar 2021 · 1 repository · arXiv:2103.11408
-
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers 15 Feb 2021 · 0 repositories · arXiv:2102.07346
-
indicnlp@kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages 14 Feb 2021 · 1 repository · arXiv:2102.07150
-
indicnlp@ kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages 14 Feb 2021 · 1 repository
-
Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers 9 Feb 2021 · 0 repositories · arXiv:2102.04697
-
Experimental Evaluation of Deep Learning models for Marathi Text Classification 13 Jan 2021 · 0 repositories · arXiv:2101.04899
-
LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT 13 Jan 2021 · 1 repository · arXiv:2101.04965
-
Post-Training Weighted Quantization of Neural Networks for Language Models 1 Jan 2021 · 0 repositories
-
HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection 1 Dec 2020 · 2 repositories
-
Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating 1 Dec 2020 · 0 repositories
-
Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification 18 Nov 2020 · 0 repositories · arXiv:2011.09448
-
Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language 13 Oct 2020 · 2 repositories · arXiv:2010.06447
-
Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection 5 Oct 2020 · 0 repositories · arXiv:2010.02094
-
Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication 12 Sep 2020 · 0 repositories · arXiv:2009.05836
-
HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection 22 Aug 2020 · 2 repositories · arXiv:2008.09820
Tasks archive 2025-07-28
20 shown of 76 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections