{"url":"/method/fastpitch","slug":"fastpitch","name":"FastPitch","full_name":"FastPitch","full_name_withheld":false,"description_markdown":"**FastPitch** is a fully-parallel text-to-speech model based on FastSpeech, conditioned on fundamental frequency contours. The architecture of FastPitch is shown in the Figure. It is based on FastSpeech and composed mainly of two feed-forward [Transformer](https://paperswithcode.com/method/transformer) (FFTr) stacks. The first one operates in the resolution of input tokens, the second one in the resolution of the output frames. Let $x=\\left(x\\_{1}, \\ldots, x\\_{n}\\right)$ be the sequence of input lexical units, and $\\mathbf{y}=\\left(y\\_{1}, \\ldots, y\\_{t}\\right)$ be the sequence of target mel-scale spectrogram frames. The first FFTr stack produces the hidden representation $\\mathbf{h}=\\operatorname{FFTr}(\\mathbf{x})$. The hidden representation $h$ is used to make predictions about the duration and average pitch of every character with a 1-D CNN \r\n\r\n$$\r\n\\hat{\\mathbf{d}}=\\text { DurationPredictor }(\\mathbf{h}), \\quad \\hat{\\mathbf{p}}=\\operatorname{PitchPredictor}(\\mathbf{h})\r\n$$\r\n\r\nwhere $\\hat{\\mathbf{d}} \\in \\mathbb{N}^{n}$ and $\\hat{\\mathbf{p}} \\in \\mathbb{R}^{n}$. Next, the pitch is projected to match the dimensionality of the hidden representation $h \\in$ $\\mathbb{R}^{n \\times d}$ and added to $\\mathbf{h}$. The resulting sum $\\mathbf{g}$ is discretely upsampled and passed to the output FFTr, which produces the output mel-spectrogram sequence\r\n\r\n$$\r\n\\mathbf{g}=\\mathbf{h}+\\operatorname{PitchEmbedding}(\\mathbf{p})\r\n$$\r\n\r\n$$\r\n\\hat{\\mathbf{y}}=\\operatorname{FFTr}\\left([\\underbrace{g\\_{1}, \\ldots, g\\_{1}}\\_{d\\_{1}}, \\ldots \\underbrace{g\\_{n}, \\ldots, g\\_{n}}_{d\\_{n}}]\\right)\r\n$$\r\n\r\n\r\nGround truth $\\mathbf{p}$ and $\\mathbf{d}$ are used during training, and predicted $\\hat{\\mathbf{p}}$ and $\\hat{\\mathbf{d}}$ are used during inference. The model optimizes mean-squared error (MSE) between the predicted and ground-truth modalities\r\n\r\n$$\r\n\\mathcal{L}=\\|\\hat{\\mathbf{y}}-\\mathbf{y}\\|\\_{2}^{2}+\\alpha\\|\\hat{\\mathbf{p}}-\\mathbf{p}\\|\\_{2}^{2}+\\gamma\\|\\hat{\\mathbf{d}}-\\mathbf{d}\\|\\_{2}^{2}\r\n$$","description_state":"present","introduced_year":null,"introduced_by":{"title":"FastPitch: Parallel Text-to-speech with Pitch Prediction","paper":"/paper/fastpitch-parallel-text-to-speech-with-pitch","first_author":"Adrian Łańcucki","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/fastpitch-parallel-text-to-speech-with-pitch"},"source":{"url":"https://arxiv.org/abs/2006.06873v2","title":"FastPitch: Parallel Text-to-speech with Pitch Prediction","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Audio","area_id":"audio","collection":"Text-to-Speech Models","url":"/methods/category/text-to-speech-models","pwc_aliases":[]}],"n_papers_tagged":9,"archive_num_papers":9,"papers_newest_first":[{"paper":null,"title":"Efficient training strategies for natural sounding speech synthesis and speaker adaptation based on FastPitch","date":"2024-10-09","arxiv_id":"2410.06787","n_code_links":0,"syntology":null},{"paper":null,"title":"Attempt Towards Stress Transfer in Speech-to-Speech Machine Translation","date":"2024-03-07","arxiv_id":"2403.04178","n_code_links":0,"syntology":null},{"paper":null,"title":"Incremental FastPitch: Chunk-based High Quality Text to Speech","date":"2024-01-03","arxiv_id":"2401.01755","n_code_links":0,"syntology":null},{"paper":"/paper/improved-child-text-to-speech-synthesis","title":"Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learning","date":"2023-11-07","arxiv_id":"2311.04313","n_code_links":1,"syntology":null},{"paper":"/paper/towards-building-text-to-speech-systems-for","title":"Towards Building Text-To-Speech Systems for the Next Billion Users","date":"2022-11-17","arxiv_id":"2211.09536","n_code_links":2,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Enhancement of Pitch Controllability using Timbre-Preserving Pitch Augmentation in FastPitch","date":"2022-04-12","arxiv_id":"2204.05753","n_code_links":0,"syntology":null},{"paper":"/paper/generative-modeling-for-low-dimensional","title":"Generative Modeling for Low Dimensional Speech Attributes with Neural Spline Flows","date":"2022-03-03","arxiv_id":"2203.01786","n_code_links":1,"syntology":null},{"paper":null,"title":"Exploring Timbre Disentanglement in Non-Autoregressive Cross-Lingual Text-to-Speech","date":"2021-10-14","arxiv_id":"2110.07192","n_code_links":0,"syntology":null},{"paper":"/paper/fastpitch-parallel-text-to-speech-with-pitch","title":"FastPitch: Parallel Text-to-speech with Pitch Prediction","date":"2020-06-11","arxiv_id":"2006.06873","n_code_links":6,"syntology":{"ran":4,"of":9,"unverified":5,"pointer_only":0}}],"papers_shown":9,"tasks":[{"task":"/task/text-to-speech","name":"Text to Speech","papers":9},{"task":"/task/text-to-speech-1","name":"text-to-speech","papers":9},{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":5},{"task":"/task/text-to-speech-synthesis","name":"Text-To-Speech Synthesis","papers":4},{"task":"/task/diversity","name":"Diversity","papers":2},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":1},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":1},{"task":"/task/disentanglement","name":"Disentanglement","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/prediction","name":"Prediction","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":1},{"task":"/task/speech-synthesis-assamese","name":"Speech Synthesis - Assamese","papers":1},{"task":"/task/speech-synthesis-bengali","name":"Speech Synthesis - Bengali","papers":1},{"task":"/task/speech-synthesis-bodo","name":"Speech Synthesis - Bodo","papers":1},{"task":"/task/speech-synthesis-gujarati","name":"Speech Synthesis - Gujarati","papers":1},{"task":"/task/speech-synthesis-hindi","name":"Speech Synthesis - Hindi","papers":1},{"task":"/task/speech-synthesis-kannada","name":"Speech Synthesis - Kannada","papers":1},{"task":"/task/speech-synthesis-malayalam","name":"Speech Synthesis - Malayalam","papers":1},{"task":"/task/speech-synthesis-manipuri","name":"Speech Synthesis - Manipuri","papers":1}],"tasks_shown":20,"n_tasks":28,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":1},{"year":"2022","papers":3},{"year":"2023","papers":1},{"year":"2024","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/fastpitch"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}