{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spirit-lm-interleaved-spoken-and-written","title":"Spirit LM: Interleaved Spoken and Written Language Model","arxiv_id":"2402.05755","date":"2024-02-08","proceeding":null,"authors":["Tu Anh Nguyen","Benjamin Muller","Bokai Yu","Marta R. Costa-Jussa","Maha Elbayad","Sravya Popuri","Christophe Ropers","Paul-Ambroise Duquenne","Robin Algayres","Ruslan Mavlyutov","Itai Gat","Mary Williamson","Gabriel Synnaeve","Juan Pino","Benoit Sagot","Emmanuel Dupoux"],"abstract":"We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code.","url_abs":"https://arxiv.org/abs/2402.05755v2","url_pdf":"https://arxiv.org/pdf/2402.05755v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spirit-lm-interleaved-spoken-and-written","repo_url":"https://github.com/facebookresearch/spiritlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-2000-hub5-english","task":"Language Modelling","dataset":"2000 HUB5 English","model":"MMLU","rank_in_archive_order":1,"of":1,"metrics":{"10-stage average accuracy":"10"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-salmon","task":"Language Modelling","dataset":"SALMon","model":"Spirit-LM (Expr.)","rank_in_archive_order":1,"of":10,"metrics":{"Background (Domain) Consistency":"55.0","Background (Random) Consistency":"64.0","Background Alignment":"59.5","Gender Consistency":"85.0","Room Consistency":"54.5","Sentiment Alignment":"52.0","Sentiment Consistency":"73.5","Speaker Consistency":"81.0"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-salmon","task":"Language Modelling","dataset":"SALMon","model":"Spirit-LM (base)","rank_in_archive_order":9,"of":10,"metrics":{"Background (Domain) Consistency":"53.5","Background (Random) Consistency":"55.5","Background Alignment":"51.5","Gender Consistency":"67.0","Room Consistency":"54.5","Sentiment Alignment":"48.0","Sentiment Consistency":"54.5","Speaker Consistency":"69.5"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/spiritlm","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"8687b6ac8f6c5823","entry":"load_pred","repo":"facebookresearch/spiritlm","repo_kind":"official","path":"spiritlm/eval/eval_stsp.py","file_url":"https://github.com/facebookresearch/spiritlm/blob/HEAD/spiritlm/eval/eval_stsp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8687b6ac8f6c5823"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}