{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/voicebox-text-guided-multilingual-universal","title":"Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale","arxiv_id":"2306.15687","date":"2023-06-23","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \\url{https://voicebox.metademolab.com}.","url_abs":"https://arxiv.org/abs/2306.15687v2","url_pdf":"https://arxiv.org/pdf/2306.15687v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"voicebox-text-guided-multilingual-universal","repo_url":"https://github.com/lucidrains/voicebox-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"speech-synthesis","task_name":"Speech Synthesis"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"text-to-speech-synthesis","task_name":"Text-To-Speech Synthesis"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[{"method_slug":"alibi","method_name":"ALiBi"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.15687","atlas_url":"https://app.syntology.ai/?focus=2306.15687","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15687"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/voicebox-pytorch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":3,"ran":3,"ran_violates":3,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f70dcb896366697c","entry":"cast_tuple","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/data.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/data.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f70dcb896366697c"}},{"code_sha256_prefix":"5cfcc1d3230706ff","entry":"cast_tuple","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/trainer.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/trainer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cfcc1d3230706ff"}},{"code_sha256_prefix":"0e0470ad1da271f6","entry":"collate_one_or_multiple_tensors","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/data.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e0470ad1da271f6"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/attend.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/attend.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/attend.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/attend.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"9354ff16b3840873","entry":"get_optimizer","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/optimizer.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9354ff16b3840873"}},{"code_sha256_prefix":"f3232418205f7cbd","entry":"identity","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/voicebox_pytorch.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/voicebox_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3232418205f7cbd"}},{"code_sha256_prefix":"52e10a7d6b0fa036","entry":"once","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/attend.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/attend.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52e10a7d6b0fa036"}},{"code_sha256_prefix":"fdc11aefc97f3abb","entry":"separate_weight_decayable_params","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/optimizer.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc11aefc97f3abb"}},{"code_sha256_prefix":"d95562c398a6b25e","entry":"yes_or_no","repo":"lucidrains/voicebox-pytorch","repo_kind":"listed","path":"voicebox_pytorch/trainer.py","file_url":"https://github.com/lucidrains/voicebox-pytorch/blob/HEAD/voicebox_pytorch/trainer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d95562c398a6b25e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}