{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/naturalspeech-2-latent-diffusion-models-are","title":"NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers","arxiv_id":"2304.09116","date":"2023-04-18","proceeding":null,"authors":["Kai Shen","Zeqian Ju","Xu Tan","Yanqing Liu","Yichong Leng","Lei He","Tao Qin","Sheng Zhao","Jiang Bian"],"abstract":"Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.","url_abs":"https://arxiv.org/abs/2304.09116v3","url_pdf":"https://arxiv.org/pdf/2304.09116v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"naturalspeech-2-latent-diffusion-models-are","repo_url":"https://github.com/adelacvg/ns2vc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"naturalspeech-2-latent-diffusion-models-are","repo_url":"https://github.com/lucidrains/naturalspeech2-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"speech-synthesis","task_name":"Speech Synthesis"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.09116","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.09116"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/naturalspeech2-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adelacvg/ns2vc","reach":{"status":"ok"}}],"summary":{"ran_violates":5,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"listed":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"d1ef6b8cb9a28a53","entry":"default","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/naturalspeech2_pytorch.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/naturalspeech2_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1ef6b8cb9a28a53"}},{"code_sha256_prefix":"f912c209e9c899ca","entry":"divisible_by","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/naturalspeech2_pytorch.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/naturalspeech2_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f912c209e9c899ca"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/aligner.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/aligner.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"608e364a9d2376a3","entry":"exists","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/naturalspeech2_pytorch.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/naturalspeech2_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"608e364a9d2376a3"}},{"code_sha256_prefix":"52e10a7d6b0fa036","entry":"once","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/attend.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/attend.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52e10a7d6b0fa036"}},{"code_sha256_prefix":"040fb4189642c954","entry":"average_over_durations","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/utils/utils.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"040fb4189642c954"}},{"code_sha256_prefix":"ac1731a11eef3564","entry":"create_mask","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/utils/utils.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac1731a11eef3564"}},{"code_sha256_prefix":"a57ad0dfe43eabea","entry":"is_tool","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/utils/phonemizers/espeak_wrapper.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/utils/phonemizers/espeak_wrapper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a57ad0dfe43eabea"}},{"code_sha256_prefix":"bfae004145c18b24","entry":"maximum_path","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/aligner.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/aligner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfae004145c18b24"}},{"code_sha256_prefix":"c6289707dcc4058f","entry":"pad_tensor","repo":"lucidrains/naturalspeech2-pytorch","repo_kind":"listed","path":"naturalspeech2_pytorch/aligner.py","file_url":"https://github.com/lucidrains/naturalspeech2-pytorch/blob/HEAD/naturalspeech2_pytorch/aligner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6289707dcc4058f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}