Papers › Deep Independently Recurrent Neural Network (IndRNN)
Deep Independently Recurrent Neural Network (IndRNN)
Shuai Li, Wanqing Li, Chris Cook, Yanbo Gao
Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | Penn Treebank (Character Level) | Dense IndRNN | Bit per Character (BPC) | 1.18 | #8 of 20 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Word Level) | Dense IndRNN+dynamic eval | Test perplexity | 50.97 | #13 of 43 | Archive leaderboard | report |
| Language Modelling | Penn Treebank (Word Level) | Dense IndRNN | Test perplexity | 56.37 | #27 of 43 | Archive leaderboard | report |
| Sequential Image Classification | Sequential MNIST | Dense IndRNN | Permuted Accuracy | 97.2% | #12 of 30 | Archive leaderboard | report |
| Sequential Image Classification | Sequential MNIST | Dense IndRNN | Unpermuted Accuracy | 99.48% | #12 of 30 | Archive leaderboard | report |
| Skeleton Based Action Recognition | NTU RGB+D | Dense IndRNN | Accuracy (CS) | 86.70 | #81 of 135 | Archive leaderboard | report |
| Skeleton Based Action Recognition | NTU RGB+D | Dense IndRNN | Accuracy (CV) | 93.97 | #81 of 135 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections