{"url":"/method/wavernn","slug":"wavernn","name":"WaveRNN","full_name":"WaveRNN","full_name_withheld":false,"description_markdown":"**WaveRNN** is a single-layer recurrent neural network for audio generation that is designed efficiently predict 16-bit raw audio samples.\r\n\r\nThe overall computation in the WaveRNN is as follows (biases omitted for brevity):\r\n\r\n$$ \\mathbf{x}\\_{t} = \\left[\\mathbf{c}\\_{t−1},\\mathbf{f}\\_{t−1}, \\mathbf{c}\\_{t}\\right] $$\r\n\r\n$$ \\mathbf{u}\\_{t} = \\sigma\\left(\\mathbf{R}\\_{u}\\mathbf{h}\\_{t-1} + \\mathbf{I}^{*}\\_{u}\\mathbf{x}\\_{t}\\right) $$\r\n\r\n$$ \\mathbf{r}\\_{t} = \\sigma\\left(\\mathbf{R}\\_{r}\\mathbf{h}\\_{t-1} + \\mathbf{I}^{*}\\_{r}\\mathbf{x}\\_{t}\\right) $$\r\n\r\n$$ \\mathbf{e}\\_{t} = \\tau\\left(\\mathbf{r}\\_{t} \\odot \\left(\\mathbf{R}\\_{e}\\mathbf{h}\\_{t-1}\\right) + \\mathbf{I}^{*}\\_{e}\\mathbf{x}\\_{t} \\right) $$\r\n\r\n$$ \\mathbf{h}\\_{t} = \\mathbf{u}\\_{t} \\cdot \\mathbf{h}\\_{t-1} + \\left(1-\\mathbf{u}\\_{t}\\right) \\cdot \\mathbf{e}\\_{t} $$\r\n\r\n$$ \\mathbf{y}\\_{c}, \\mathbf{y}\\_{f} = \\text{split}\\left(\\mathbf{h}\\_{t}\\right) $$\r\n\r\n$$ P\\left(\\mathbf{c}\\_{t}\\right) = \\text{softmax}\\left(\\mathbf{O}\\_{2}\\text{relu}\\left(\\mathbf{O}\\_{1}\\mathbf{y}\\_{c}\\right)\\right) $$\r\n\r\n$$ P\\left(\\mathbf{f}\\_{t}\\right) = \\text{softmax}\\left(\\mathbf{O}\\_{4}\\text{relu}\\left(\\mathbf{O}\\_{3}\\mathbf{y}\\_{f}\\right)\\right) $$\r\n\r\nwhere the $*$ indicates a masked matrix whereby the last coarse input $\\mathbf{c}\\_{t}$ is only connected to the fine part of the states $\\mathbf{u}\\_{t}$, $\\mathbf{r}\\_{t}$, $\\mathbf{e}\\_{t}$ and $\\mathbf{h}\\_{t}$ and thus only affects the fine output $\\mathbf{y}\\_{f}$. The coarse and fine parts $\\mathbf{c}\\_{t}$ and $\\mathbf{f}\\_{t}$ are encoded as scalars in $\\left[0, 255\\right]$ and scaled to the interval $\\left[−1, 1\\right]$. The matrix $\\mathbf{R}$ formed from the matrices $\\mathbf{R}\\_{u}$, $\\mathbf{R}\\_{r}$, $\\mathbf{R}\\_{e}$ is computed as a single matrix-vector product to produce the contributions to all three gates $\\mathbf{u}\\_{t}$, $mathbf{r}\\_{t}$ and $\\mathbf{e}\\_{t}$ (a variant of the [GRU cell](https://paperswithcode.com/method/gru). $\\sigma$ and $\\tau$ are the standard sigmoid and tanh non-linearities.\r\n\r\nEach part feeds into a [softmax](https://paperswithcode.com/method/softmax) layer over the corresponding 8 bits and the prediction of the 8 fine bits is conditioned on the 8 coarse bits. The resulting Dual Softmax layer allows for efficient prediction of 16-bit samples using two small output spaces (2 8 values each) instead of a single large output space (with 2 16 values).","description_state":"present","introduced_year":null,"introduced_by":{"title":"Efficient Neural Audio Synthesis","paper":"/paper/efficient-neural-audio-synthesis","first_author":"Nal Kalchbrenner","n_authors":10,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/efficient-neural-audio-synthesis"},"source":{"url":"http://arxiv.org/abs/1802.08435v2","title":"Efficient Neural Audio Synthesis","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Sequential","area_id":"sequential","collection":"Recurrent Neural Networks","url":"/methods/category/recurrent-neural-networks","pwc_aliases":[]},{"area":"Audio","area_id":"audio","collection":"Generative Audio Models","url":"/methods/category/generative-audio-models","pwc_aliases":[]}],"n_papers_tagged":26,"archive_num_papers":26,"papers_newest_first":[{"paper":null,"title":"Exploratory Evaluation of Speech Content Masking","date":"2024-01-08","arxiv_id":"2401.03936","n_code_links":0,"syntology":null},{"paper":null,"title":"An End-to-End Multi-Module Audio Deepfake Generation System for ADD Challenge 2023","date":"2023-07-03","arxiv_id":"2307.00729","n_code_links":0,"syntology":null},{"paper":null,"title":"Evince the artifacts of Spoof Speech by blending Vocal Tract and Voice Source Features","date":"2022-12-05","arxiv_id":"2212.02013","n_code_links":0,"syntology":null},{"paper":null,"title":"SIMD-size aware weight regularization for fast neural vocoding on CPU","date":"2022-11-02","arxiv_id":"2211.00898","n_code_links":0,"syntology":null},{"paper":"/paper/perfectly-secure-steganography-using-minimum","title":"Perfectly Secure Steganography Using Minimum Entropy Coupling","date":"2022-10-24","arxiv_id":"2210.14889","n_code_links":2,"syntology":{"ran":0,"of":4,"unverified":4,"pointer_only":0}},{"paper":null,"title":"Adaptive re-calibration of channel-wise features for Adversarial Audio Classification","date":"2022-10-21","arxiv_id":"2210.11722","n_code_links":0,"syntology":null},{"paper":null,"title":"WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration","date":"2022-10-03","arxiv_id":"2210.01029","n_code_links":0,"syntology":null},{"paper":null,"title":"R-MelNet: Reduced Mel-Spectral Modeling for Neural TTS","date":"2022-06-30","arxiv_id":"2206.15276","n_code_links":0,"syntology":null},{"paper":null,"title":"NatiQ: An End-to-end Text-to-Speech System for Arabic","date":"2022-06-15","arxiv_id":"2206.07373","n_code_links":0,"syntology":null},{"paper":"/paper/vocbench-a-neural-vocoder-benchmark-for","title":"VocBench: A Neural Vocoder Benchmark for Speech Synthesis","date":"2021-12-06","arxiv_id":"2112.03099","n_code_links":1,"syntology":null},{"paper":null,"title":"On-device neural speech synthesis","date":"2021-09-17","arxiv_id":"2109.08710","n_code_links":0,"syntology":null},{"paper":"/paper/high-fidelity-and-low-latency-universal","title":"High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling","date":"2021-05-20","arxiv_id":"2105.09856","n_code_links":1,"syntology":null},{"paper":"/paper/low-latency-real-time-non-parallel-voice","title":"Low-Latency Real-Time Non-Parallel Voice Conversion based on Cyclic Variational Autoencoder and Multiband WaveRNN with Data-Driven Linear Prediction","date":"2021-05-20","arxiv_id":"2105.09858","n_code_links":2,"syntology":null},{"paper":null,"title":"Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders","date":"2021-02-12","arxiv_id":"2102.06610","n_code_links":0,"syntology":null},{"paper":null,"title":"FBWave: Efficient and Scalable Neural Vocoders for Streaming Text-To-Speech on the Edge","date":"2020-11-25","arxiv_id":"2011.12985","n_code_links":0,"syntology":null},{"paper":"/paper/tfgan-time-and-frequency-domain-based","title":"TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis","date":"2020-11-24","arxiv_id":"2011.12206","n_code_links":1,"syntology":null},{"paper":null,"title":"Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis","date":"2020-11-10","arxiv_id":"2011.04839","n_code_links":0,"syntology":null},{"paper":"/paper/enhancing-speech-intelligibility-in-text-to","title":"Enhancing Speech Intelligibility in Text-To-Speech Synthesis using Speaking Style Conversion","date":"2020-08-13","arxiv_id":"2008.05809","n_code_links":1,"syntology":{"ran":2,"of":3,"unverified":1,"pointer_only":0}},{"paper":"/paper/speaker-conditional-wavernn-towards-universal","title":"Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions","date":"2020-08-09","arxiv_id":"2008.05289","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":null,"title":"Audiovisual Speech Synthesis using Tacotron2","date":"2020-08-03","arxiv_id":"2008.00620","n_code_links":0,"syntology":null},{"paper":"/paper/end-to-end-speech-synthesis-applied-to","title":"TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese","date":"2020-05-11","arxiv_id":"2005.05144","n_code_links":1,"syntology":null},{"paper":null,"title":"Towards Robust Neural Vocoding for Speech Generation: A Survey","date":"2019-12-05","arxiv_id":"1912.02461","n_code_links":0,"syntology":null},{"paper":null,"title":"A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis","date":"2019-11-11","arxiv_id":"1911.04111","n_code_links":0,"syntology":null},{"paper":"/paper/durian-duration-informed-attention-network","title":"DurIAN: Duration Informed Attention Network For Multimodal Synthesis","date":"2019-09-04","arxiv_id":"1909.01700","n_code_links":6,"syntology":null},{"paper":"/paper/lpcnet-improving-neural-speech-synthesis","title":"LPCNet: Improving Neural Speech Synthesis Through Linear Prediction","date":"2018-10-28","arxiv_id":"1810.11846","n_code_links":2,"syntology":null},{"paper":"/paper/efficient-neural-audio-synthesis","title":"Efficient Neural Audio Synthesis","date":"2018-02-23","arxiv_id":"1802.08435","n_code_links":16,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":1}}],"papers_shown":26,"tasks":[{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":14},{"task":"/task/text-to-speech","name":"Text to Speech","papers":11},{"task":"/task/text-to-speech-1","name":"text-to-speech","papers":11},{"task":null,"name":"CPU","papers":5},{"task":"/task/text-to-speech-synthesis","name":"Text-To-Speech Synthesis","papers":4},{"task":null,"name":"GPU","papers":3},{"task":"/task/decoder","name":"Decoder","papers":2},{"task":"/task/denoising","name":"Denoising","papers":2},{"task":"/task/face-swapping","name":"Face Swapping","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/voice-conversion","name":"Voice Conversion","papers":2},{"task":"/task/audio-classification","name":"Audio Classification","papers":1},{"task":"/task/audio-synthesis","name":"Audio Synthesis","papers":1},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":1},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":1},{"task":"/task/edge-classification","name":"Edge Classification","papers":1},{"task":"/task/face-model","name":"Face Model","papers":1},{"task":null,"name":"Generative Adversarial Network","papers":1},{"task":"/task/low-latency-processing","name":"Low-latency processing","papers":1},{"task":"/task/polyphone-disambiguation","name":"Polyphone disambiguation","papers":1}],"tasks_shown":20,"n_tasks":27,"usage_by_year":[{"year":"2018","papers":2},{"year":"2019","papers":3},{"year":"2020","papers":7},{"year":"2021","papers":5},{"year":"2022","papers":7},{"year":"2023","papers":1},{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/wavernn"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}