{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llama-omni-seamless-speech-interaction-with","title":"LLaMA-Omni: Seamless Speech Interaction with Large Language Models","arxiv_id":"2409.06666","date":"2024-09-10","proceeding":null,"authors":["Qingkai Fang","Shoutao Guo","Yan Zhou","Zhengrui Ma","Shaolei Zhang","Yang Feng"],"abstract":"Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.","url_abs":"https://arxiv.org/abs/2409.06666v2","url_pdf":"https://arxiv.org/pdf/2409.06666v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llama-omni-seamless-speech-interaction-with","repo_url":"https://github.com/ictnlp/llama-omni","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"llama-omni-seamless-speech-interaction-with","repo_url":"https://github.com/DanielLin94144/Full-Duplex-Bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"llama-omni-seamless-speech-interaction-with","repo_url":"https://github.com/wntg/LLaMA-Omni","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.06666","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.06666"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wntg/LLaMA-Omni","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ictnlp/llama-omni","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DanielLin94144/Full-Duplex-Bench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"ran_fixture":1,"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"4c5103ee8669c997","entry":"EncoderProjectorConcat","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c5103ee8669c997"}},{"code_sha256_prefix":"4da597212fee8c5f","entry":"OmniSpeechConfig","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4da597212fee8c5f"}},{"code_sha256_prefix":"b262640dbfbc7946","entry":"build_speech_encoder","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b262640dbfbc7946"}},{"code_sha256_prefix":"cf80d0b1cc5418ca","entry":"build_speech_projector","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf80d0b1cc5418ca"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"be916bc2b37c9888","entry":"OmniSpeechLlamaForCausalLM","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be916bc2b37c9888"}},{"code_sha256_prefix":"90a4ef0aa95d7a5f","entry":"OmniSpeechLlamaForCausalLM","repo":"ictnlp/llama-omni","repo_kind":"official","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/ictnlp/llama-omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90a4ef0aa95d7a5f"}},{"code_sha256_prefix":"56462b063fec0c44","entry":"OmniSpeechLlamaModel","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56462b063fec0c44"}},{"code_sha256_prefix":"79aa45784d52b456","entry":"OmniSpeechMetaForCausalLM","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79aa45784d52b456"}},{"code_sha256_prefix":"b28bc26a5d2baca8","entry":"OmniSpeechMetaModel","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b28bc26a5d2baca8"}},{"code_sha256_prefix":"c03b4f330d42219f","entry":"WhisperWrappedEncoder","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/model/language_model/omni_speech_llama.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/model/language_model/omni_speech_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c03b4f330d42219f"}},{"code_sha256_prefix":"300383390ba66179","entry":"collate_fn","repo":"wntg/LLaMA-Omni","repo_kind":"listed","path":"omni_speech/train/stage2.py","file_url":"https://github.com/wntg/LLaMA-Omni/blob/HEAD/omni_speech/train/stage2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"300383390ba66179"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}