{"url":"/method/qrnn","slug":"qrnn","name":"QRNN","full_name":"Quasi-Recurrent Neural Network","full_name_withheld":false,"description_markdown":"A **QRNN**, or **Quasi-Recurrent Neural Network**, is a type of recurrent neural network that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Due to their increased parallelism, they can be up to 16 times faster at train and test time than [LSTMs](https://paperswithcode.com/method/lstm).\r\n\r\nGiven an input sequence $\\mathbf{X} \\in \\mathbb{R}^{T\\times{n}}$ of $T$ n-dimensional vectors $\\mathbf{x}\\_{1}, \\dots, \\mathbf{x}\\_{T}$, the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of $m$ filters, producing a sequence $\\mathbf{Z} \\in \\mathbb{R}^{T\\times{m}}$ of m-dimensional candidate vectors $\\mathbf{z}\\_{t}$. Masked convolutions are used so filters can not access information from future timesteps (implementing with left padding).\r\n\r\nAdditional convolutions are applied with separate filter banks to obtain sequences of vectors for the\r\nelementwise gates that are needed for the pooling function. While the candidate vectors are passed\r\nthrough a $\\tanh$ nonlinearity, the gates use an elementwise sigmoid. If the pooling function requires a\r\nforget gate $f\\_{t}$ and an output gate $o\\_{t}$ at each timestep, the full set of computations in the convolutional component is then:\r\n\r\n$$ \\mathbf{Z} = \\tanh\\left(\\mathbf{W}\\_{z} ∗ \\mathbf{X}\\right) $$\r\n$$ \\mathbf{F} = \\sigma\\left(\\mathbf{W}\\_{f} ∗ \\mathbf{X}\\right) $$\r\n$$ \\mathbf{O} = \\sigma\\left(\\mathbf{W}\\_{o} ∗ \\mathbf{X}\\right) $$\r\n\r\nwhere $\\mathbf{W}\\_{z}$, $\\mathbf{W}\\_{f}$, and $\\mathbf{W}\\_{o}$, each in $\\mathbb{R}^{k×n×m}$, are the convolutional filter banks and ∗ denotes a [masked convolution](https://paperswithcode.com/method/masked-convolution) along the timestep dimension.  Dynamic [average pooling](https://paperswithcode.com/method/average-pooling) by Balduzzi & Ghifary (2016) is used, which uses only a forget gate:\r\n\r\n$$ \\mathbf{h}\\_{t} = \\mathbf{f}\\_{t} \\odot{\\mathbf{h}\\_{t−1}} + \\left(1 − \\mathbf{f}\\_{t}\\right) \\odot{\f\\mathbf{z}\\_{t}} $$ \r\n\r\nWhich is denoted f-pooling. The function may also include an output gate:\r\n\r\n$$ \\mathbf{c}\\_{t} = \\mathbf{f}\\_{t} \\odot{\\mathbf{c}\\_{t−1}} + \\left(1 − \\mathbf{f}\\_{t}\\right) \\odot{\f\\mathbf{z}\\_{t}} $$ \r\n\r\n$$ \\mathbf{h}\\_{t} = \\mathbf{o}\\_{t} \\odot{\\mathbf{c}\\_{t}} $$\r\n\r\nWhich is denoted fo-pooling. Or the recurrence relation may include an independent input and forget gate:\r\n\r\n$$ \\mathbf{c}\\_{t} = \\mathbf{f}\\_{t} \\odot{\\mathbf{c}\\_{t−1}} + \\mathbf{i}\\_{t}\\odot{\f\\mathbf{z}\\_{t}} $$ \r\n\r\n$$ \\mathbf{h}\\_{t} = \\mathbf{o}\\_{t} \\odot{\\mathbf{c}\\_{t}} $$\r\n\r\nWhich is denoted ifo-pooling. In each case $h$ or $c$ is initialized to zero. The recurrent part sof these functions must be calculated for each timestep in the sequence, but parallelism along feature dimensions means evaluating them even over long sequences requires a negligible amount of computation time.\r\n\r\nA single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features. As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://arxiv.org/abs/1611.01576v2","title":"Quasi-Recurrent Neural Networks","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Sequential","area_id":"sequential","collection":"Recurrent Neural Networks","url":"/methods/category/recurrent-neural-networks","pwc_aliases":[]}],"n_papers_tagged":15,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversion","date":"2024-08-18","arxiv_id":"2408.09403","n_code_links":0,"syntology":null},{"paper":null,"title":"Ensemble learning for predictive uncertainty estimation with application to the correction of satellite precipitation products","date":"2024-03-14","arxiv_id":"2403.10567","n_code_links":0,"syntology":null},{"paper":null,"title":"Uncertainty estimation of machine learning spatial precipitation predictions from satellite data","date":"2023-11-13","arxiv_id":"2311.07511","n_code_links":0,"syntology":null},{"paper":null,"title":"Density Matrix Emulation of Quantum Recurrent Neural Networks for Multivariate Time Series Prediction","date":"2023-10-31","arxiv_id":"2310.20671","n_code_links":0,"syntology":null},{"paper":null,"title":"Efficient quantum recurrent reinforcement learning via quantum reservoir computing","date":"2023-09-13","arxiv_id":"2309.07339","n_code_links":0,"syntology":null},{"paper":"/paper/quantum-recurrent-neural-networks-for","title":"Quantum Recurrent Neural Networks for Sequential Learning","date":"2023-02-07","arxiv_id":"2302.03244","n_code_links":1,"syntology":null},{"paper":null,"title":"Time-Warping Invariant Quantum Recurrent Neural Networks via Quantum-Classical Adaptive Gating","date":"2023-01-19","arxiv_id":"2301.08173","n_code_links":0,"syntology":null},{"paper":null,"title":"Reservoir Computing via Quantum Recurrent Neural Networks","date":"2022-11-04","arxiv_id":"2211.02612","n_code_links":0,"syntology":null},{"paper":"/paper/rapid-training-of-quantum-recurrent-neural","title":"Rapid training of quantum recurrent neural networks","date":"2022-07-01","arxiv_id":"2207.00378","n_code_links":1,"syntology":null},{"paper":"/paper/tyolov5-a-temporal-yolov5-detector-based-on","title":"TYolov5: A Temporal Yolov5 Detector Based on Quasi-Recurrent Neural Networks for Real-Time Handgun Detection in Video","date":"2021-11-17","arxiv_id":"2111.08867","n_code_links":1,"syntology":null},{"paper":null,"title":"Cyber Threat Intelligence for Secure Smart City","date":"2020-07-26","arxiv_id":"2007.13233","n_code_links":0,"syntology":null},{"paper":null,"title":"A Formal Hierarchy of RNN Architectures","date":"2020-04-18","arxiv_id":"2004.08500","n_code_links":0,"syntology":null},{"paper":null,"title":"Single Stream Parallelization of Recurrent Neural Networks for Low Power and Fast Inference","date":"2018-03-30","arxiv_id":"1803.11389","n_code_links":0,"syntology":null},{"paper":"/paper/dual-rectified-linear-units-drelus-a","title":"Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks","date":"2017-07-25","arxiv_id":"1707.08214","n_code_links":2,"syntology":null},{"paper":"/paper/quasi-recurrent-neural-networks","title":"Quasi-Recurrent Neural Networks","date":"2016-11-05","arxiv_id":"1611.01576","n_code_links":7,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}}],"papers_shown":15,"tasks":[{"task":"/task/time-series-prediction","name":"Time Series Prediction","papers":3},{"task":"/task/decision-making","name":"Decision Making","papers":2},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":2},{"task":"/task/sentiment-classification","name":"Sentiment Classification","papers":2},{"task":"/task/spatial-interpolation","name":"Spatial Interpolation","papers":2},{"task":"/task/time-series-1","name":"Time Series","papers":2},{"task":"/task/time-series","name":"Time Series Analysis","papers":2},{"task":"/task/quantile-regression","name":"quantile regression","papers":2},{"task":"/task/regression-1","name":"regression","papers":2},{"task":"/task/benchmarking","name":"Benchmarking","papers":1},{"task":null,"name":"CPU","papers":1},{"task":"/task/collision-avoidance","name":"Collision Avoidance","papers":1},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":1},{"task":"/task/ensemble-learning","name":"Ensemble Learning","papers":1},{"task":"/task/feature-engineering","name":"Feature Engineering","papers":1},{"task":"/task/feature-importance","name":"Feature Importance","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-augmentation","name":"Image Augmentation","papers":1}],"tasks_shown":20,"n_tasks":34,"usage_by_year":[{"year":"2016","papers":1},{"year":"2017","papers":1},{"year":"2018","papers":1},{"year":"2020","papers":2},{"year":"2021","papers":1},{"year":"2022","papers":2},{"year":"2023","papers":5},{"year":"2024","papers":2}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/qrnn"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}