Papers › Improved Language Modeling by Decoding the Past
Improved Language Modeling by Decoding the Past
Siddhartha Brahma
Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token. This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | Penn Treebank (Character Level) | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Bit per Character (BPC) | 1.169 | #6 of 20 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Character Level) | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Number of params | 13.8M | #6 of 20 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Word Level) | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Params | 22M | #9 of 43 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Word Level) | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Test perplexity | 47.3 | #9 of 43 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Word Level) | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Validation perplexity | 48.0 | #9 of 43 | Archive leaderboard | report |
| Language Modelling | WikiText-2 | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Number of params | 35M | #14 of 38 | Archive leaderboard | report |
| Language Modelling | WikiText-2 | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Test perplexity | 40.3 | #14 of 38 | Archive leaderboard | report |
| Language Modelling | WikiText-2 | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | Validation perplexity | 42.0 | #14 of 38 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections