Papers › Recurrent Highway Networks with Grouped Auxiliary Memory

Recurrent Highway Networks with Grouped Auxiliary Memory

13 Dec 2019IEEE Access 2019 12archive 2025-07-28

Wei Luo ; Feng Yu

Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger step-to-step transition depth, leading to more expressive models. However, problems that require capturing long-term dependencies still can not be well addressed by these models. Moreover, the ability to keep long-term memories tends to diminish when the spatial depth increases, since deeper structure may accelerate gradient vanishing. In this paper, we address these issues by proposing a novel RNN architecture based on RHN, namely the Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN). The proposed architecture interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, which is analogous to Memory Augmented Neural Networks (MANNs). Experimental results on artificial long time lag tasks show that GAM-RHNs can be trained efficiently while being deep in both time and space. We also evaluate the proposed architecture on a variety of tasks, including language modeling, sequential image classification, and financial market forecasting. The potential of our approach is demonstrated by achieving state-of-the-art results on these tasks.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

MindCode-4/code-13 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning (arXiv:2212.08663; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
MindCode-4/code-8 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations (arXiv:2103.13843; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
MindSpore-scientific/code-5 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Similarity of Neural Network Models: A Survey of Functional and Representational Measures (arXiv:2305.06329; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
WilliamRo/gam_rhn tfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationLanguage ModelingLanguage ModellingSequential Image ClassificationStock Trend Predictionimage-classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Language Modelling Penn Treebank (Character Level) GAM-RHN-5 Bit per Character (BPC) 1.147 #3 of 20 Archive leaderboard report
Language Modelling Penn Treebank (Character Level) GAM-RHN-5 Number of params 16.0M #3 of 20 Archive leaderboard report
Language Modelling Text8 GAM-RHN-10 Bit per Character (BPC) 1.157 #12 of 24 Archive leaderboard report
Language Modelling Text8 GAM-RHN-10 Number of params 44.7M #12 of 24 Archive leaderboard report
Sequential Image Classification Sequential MNIST GAM-RHN-1 Permuted Accuracy 96.8% #17 of 30 Archive leaderboard report
Stock Trend Prediction FI-2010 BL-GAM-RHN-7 Accuracy (H50) 0.8202 #1 of 1 Archive leaderboard report
Stock Trend Prediction FI-2010 BL-GAM-RHN-7 F1 (H50) 0.8088 #1 of 1 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Highway LayerHighway NetworkSigmoid Activation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections