{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fish-speech-leveraging-large-language-models","title":"Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis","arxiv_id":"2411.01156","date":"2024-11-02","proceeding":null,"authors":["Shijia Liao","Yuxuan Wang","Tianyu Li","Yifan Cheng","Ruoyi Zhang","Rongzhi Zhou","Yijin Xing"],"abstract":"Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100\\% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at \\href{https://github.com/fishaudio/fish-speech}{https://github.com/fishaudio/fish-speech}.","url_abs":"https://arxiv.org/abs/2411.01156v2","url_pdf":"https://arxiv.org/pdf/2411.01156v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"fish-speech-leveraging-large-language-models","repo_url":"https://github.com/fishaudio/fish-speech","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.01156","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01156"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fishaudio/fish-speech","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"6bf15508c4c6d6e9","entry":"extract_i18n_strings","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/i18n/scan.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/i18n/scan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6bf15508c4c6d6e9"}},{"code_sha256_prefix":"fbf6eb533e5e6463","entry":"get_constant_schedule_with_warmup_lr_lambda","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/scheduler.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fbf6eb533e5e6463"}},{"code_sha256_prefix":"fd4c3ecaac2f0591","entry":"get_cosine_schedule_with_warmup_lr_lambda","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/scheduler.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fd4c3ecaac2f0591"}},{"code_sha256_prefix":"0eb12dee5313b7f7","entry":"grad_norm","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/callbacks/grad_norm.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/callbacks/grad_norm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0eb12dee5313b7f7"}},{"code_sha256_prefix":"ea6cf2a98f0b4c5b","entry":"load_language_list","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/i18n/core.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/i18n/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ea6cf2a98f0b4c5b"}},{"code_sha256_prefix":"3cef1fb59e884f2b","entry":"restore_ndarray","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/content_sequence.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/content_sequence.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3cef1fb59e884f2b"}},{"code_sha256_prefix":"23bdd89ac9ca906b","entry":"split_by_rank_worker","repo":"fishaudio/fish-speech","repo_kind":"official","path":"fish_speech/datasets/semantic.py","file_url":"https://github.com/fishaudio/fish-speech/blob/HEAD/fish_speech/datasets/semantic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"23bdd89ac9ca906b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}