Papers › Recurrent Highway Networks with Grouped Auxiliary Memory
Recurrent Highway Networks with Grouped Auxiliary Memory
Wei Luo ; Feng Yu
Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger step-to-step transition depth, leading to more expressive models. However, problems that require capturing long-term dependencies still can not be well addressed by these models. Moreover, the ability to keep long-term memories tends to diminish when the spatial depth increases, since deeper structure may accelerate gradient vanishing. In this paper, we address these issues by proposing a novel RNN architecture based on RHN, namely the Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN). The proposed architecture interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, which is analogous to Memory Augmented Neural Networks (MANNs). Experimental results on artificial long time lag tasks show that GAM-RHNs can be trained efficiently while being deep in both time and space. We also evaluate the proposed architecture on a variety of tasks, including language modeling, sequential image classification, and financial market forecasting. The potential of our approach is demonstrated by achieving state-of-the-art results on these tasks.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | Penn Treebank (Character Level) | GAM-RHN-5 | Bit per Character (BPC) | 1.147 | #3 of 20 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Character Level) | GAM-RHN-5 | Number of params | 16.0M | #3 of 20 | Archive leaderboard | report |
| Language Modelling | Text8 | GAM-RHN-10 | Bit per Character (BPC) | 1.157 | #12 of 24 | Archive leaderboard | report |
| Language Modelling | Text8 | GAM-RHN-10 | Number of params | 44.7M | #12 of 24 | Archive leaderboard | report |
| Sequential Image Classification | Sequential MNIST | GAM-RHN-1 | Permuted Accuracy | 96.8% | #17 of 30 | Archive leaderboard | report |
| Stock Trend Prediction | FI-2010 | BL-GAM-RHN-7 | Accuracy (H50) | 0.8202 | #1 of 1 | Archive leaderboard | report |
| Stock Trend Prediction | FI-2010 | BL-GAM-RHN-7 | F1 (H50) | 0.8088 | #1 of 1 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections