Papers › Memory-efficient Stochastic methods for Memory-based Transformers
Memory-efficient Stochastic methods for Memory-based Transformers
Vishwajit Kumar Vishnu, C. Chandra Sekhar
Training Memory-based transformers can require a large amount of memory and can be quite inefficient. We propose a novel two-phase training mechanism and a novel regularization technique to improve the training efficiency of memory-based transformers, which are often used for long-range context problems. For our experiments, we consider transformer-XL as our baseline model which is one of memorybased transformer models. We show that our resultant model, Skip Cross-head TransformerXL, outperforms the baseline on character level language modeling task with similar parameters and outperforms the baseline on word level language modelling task with almost 20% fewer parameters. Our proposed methods do not require any additional memory. We also demonstrate the effectiveness of our regularization mechanism on BERT which shows similar performance with reduction in standard deviation of scores of around 30% on multiple GLUE tasks.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | WikiText-103 | Skip Cross-Head Transformer-XL | Number of params | 122M | #49 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Skip Cross-Head Transformer-XL | Test perplexity | 22.91 | #49 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Skip Cross-Head Transformer-XL | Validation perplexity | 21.87 | #49 of 89 | Archive leaderboard | report |
| Language Modelling | enwik8 | Skip Cross-Head Transformer-XL | Bit per Character (BPC) | 1.033 | #24 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | Skip Cross-Head Transformer-XL | Number of params | 41M | #24 of 42 | Archive leaderboard | report |
| Paraphrase Identification | Quora Question Pairs Dev | BERT + SCH attm | Val Accuracy | 91.422 | #1 of 2 | Archive leaderboard | report |
| Paraphrase Identification | Quora Question Pairs Dev | BERT + SCH attn | Val F1 Score | 88.436 | #2 of 2 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections