{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/funaudiollm-voice-understanding-and","title":"FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs","arxiv_id":"2407.04051","date":"2024-07-04","proceeding":null,"authors":["Keyu An","Qian Chen","Chong Deng","Zhihao Du","Changfeng Gao","Zhifu Gao","Yue Gu","Ting He","Hangrui Hu","Kai Hu","Shengpeng Ji","Yabin Li","Zerui Li","Heng Lu","Haoneng Luo","Xiang Lv","Bin Ma","Ziyang Ma","Chongjia Ni","Changhe Song","Jiaqi Shi","Xian Shi","Hao Wang","Wen Wang","Yuxuan Wang","Zhangyu Xiao","Zhijie Yan","Yexin Yang","Bin Zhang","Qinglin Zhang","Shiliang Zhang","Nan Zhao","Siqi Zheng"],"abstract":"This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.","url_abs":"https://arxiv.org/abs/2407.04051v3","url_pdf":"https://arxiv.org/pdf/2407.04051v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"funaudiollm-voice-understanding-and","repo_url":"https://github.com/FunAudioLLM/SenseVoice","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"funaudiollm-voice-understanding-and","repo_url":"https://github.com/funaudiollm/cosyvoice","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"funaudiollm-voice-understanding-and","repo_url":"https://github.com/xingchensong/s3tokenizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"event-detection","task_name":"Event Detection"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-to-speech-translation","task_name":"Speech-to-Speech Translation"},{"task_slug":"voice-cloning","task_name":"Voice Cloning"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.04051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.04051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/funaudiollm/cosyvoice","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingchensong/s3tokenizer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FunAudioLLM/SenseVoice","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_honours":1},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"be9e79b09e9b1c3f","entry":"create_sensevoice_model","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"long_audio_no_vad.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/long_audio_no_vad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be9e79b09e9b1c3f"}},{"code_sha256_prefix":"45fa8b136cb3c441","entry":"export_dummy_inputs","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"export_meta.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/export_meta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45fa8b136cb3c441"}},{"code_sha256_prefix":"19aab5f362ccf5f0","entry":"export_forward","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"export_meta.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/export_meta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19aab5f362ccf5f0"}},{"code_sha256_prefix":"df46a8419573e3a7","entry":"format_str","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"webui.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/webui.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df46a8419573e3a7"}},{"code_sha256_prefix":"8454be6adc9ddaa4","entry":"format_str_v2","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"webui.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/webui.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8454be6adc9ddaa4"}},{"code_sha256_prefix":"5565f8d59ef72ba2","entry":"format_str_v3","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"webui.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/webui.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5565f8d59ef72ba2"}},{"code_sha256_prefix":"51218edc823a6583","entry":"load_audio_cpp_model_spec","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"runtime/llama.cpp/export_sensevoice_gguf.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/runtime/llama.cpp/export_sensevoice_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51218edc823a6583"}},{"code_sha256_prefix":"eb667117f53a99ce","entry":"merge_transcripts","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"long_audio_no_vad.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/long_audio_no_vad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb667117f53a99ce"}},{"code_sha256_prefix":"8c9161f8d6f989d3","entry":"sequence_mask","repo":"FunAudioLLM/SenseVoice","repo_kind":"official","path":"model.py","file_url":"https://github.com/FunAudioLLM/SenseVoice/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c9161f8d6f989d3"}},{"code_sha256_prefix":"e529c9641c178fe0","entry":"sinusoids","repo":"xingchensong/s3tokenizer","repo_kind":"listed","path":"s3tokenizer/model.py","file_url":"https://github.com/xingchensong/s3tokenizer/blob/HEAD/s3tokenizer/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e529c9641c178fe0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}