{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mobilespeech-a-fast-and-high-fidelity","title":"MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech","arxiv_id":"2402.09378","date":"2024-02-14","proceeding":null,"authors":["Shengpeng Ji","Ziyue Jiang","Hanting Wang","Jialong Zuo","Zhou Zhao"],"abstract":"Zero-shot text-to-speech (TTS) has gained significant attention due to its powerful voice cloning capabilities, requiring only a few seconds of unseen speaker voice prompts. However, all previous work has been developed for cloud-based systems. Taking autoregressive models as an example, although these approaches achieve high-fidelity voice cloning, they fall short in terms of inference speed, model size, and robustness. Therefore, we propose MobileSpeech, which is a fast, lightweight, and robust zero-shot text-to-speech system based on mobile devices for the first time. Specifically: 1) leveraging discrete codec, we design a parallel speech mask decoder module called SMD, which incorporates hierarchical information from the speech codec and weight mechanisms across different codec layers during the generation process. Moreover, to bridge the gap between text and speech, we introduce a high-level probabilistic mask that simulates the progression of information flow from less to more during speech generation. 2) For speaker prompts, we extract fine-grained prompt duration from the prompt speech and incorporate text, prompt speech by cross attention in SMD. We demonstrate the effectiveness of MobileSpeech on multilingual datasets at different levels, achieving state-of-the-art results in terms of generating speed and speech quality. MobileSpeech achieves RTF of 0.09 on a single A100 GPU and we have successfully deployed MobileSpeech on mobile devices. Audio samples are available at \\url{https://mobilespeech.github.io/} .","url_abs":"https://arxiv.org/abs/2402.09378v2","url_pdf":"https://arxiv.org/pdf/2402.09378v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"voice-cloning","task_name":"Voice Cloning"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.09378","atlas_url":"https://app.syntology.ai/?focus=2402.09378","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09378"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/enhuiz/vall-e","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Edresson/YourTTS","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"ran_fixture":1,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ec7cdfe06cf76a22","entry":"AdaLN","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec7cdfe06cf76a22"}},{"code_sha256_prefix":"8fe53745e4d5d22f","entry":"Attention","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fe53745e4d5d22f"}},{"code_sha256_prefix":"4d47845809fd8661","entry":"Block","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d47845809fd8661"}},{"code_sha256_prefix":"ccb069c6188deb3c","entry":"Embedding","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccb069c6188deb3c"}},{"code_sha256_prefix":"b1cc06b94bee5ada","entry":"MultiEmbedding","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1cc06b94bee5ada"}},{"code_sha256_prefix":"b8b6021719fa7eed","entry":"PrenormResidual","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8b6021719fa7eed"}},{"code_sha256_prefix":"91e12b0e6dfa4060","entry":"SinusodialEmbedding","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91e12b0e6dfa4060"}},{"code_sha256_prefix":"d59002e01a0b0518","entry":"_create_mask","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d59002e01a0b0518"}},{"code_sha256_prefix":"f47e5f0f909741dd","entry":"_join","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f47e5f0f909741dd"}},{"code_sha256_prefix":"b06aa768d8e2dd43","entry":"list_to_tensor","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b06aa768d8e2dd43"}},{"code_sha256_prefix":"894ecc1799b226be","entry":"AR","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"894ecc1799b226be"}},{"code_sha256_prefix":"3c60927a0bd1494e","entry":"Base","repo":"enhuiz/vall-e","repo_kind":"found_in_text","path":"vall_e/vall_e/ar.py","file_url":"https://github.com/enhuiz/vall-e/blob/HEAD/vall_e/vall_e/ar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c60927a0bd1494e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}