{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/textually-pretrained-speech-language-models","title":"Textually Pretrained Speech Language Models","arxiv_id":"2305.13009","date":"2023-05-22","proceeding":"NeurIPS 2023 11","authors":["Michael Hassid","Tal Remez","Tu Anh Nguyen","Itai Gat","Alexis Conneau","Felix Kreuk","Jade Copet","Alexandre Defossez","Gabriel Synnaeve","Emmanuel Dupoux","Roy Schwartz","Yossi Adi"],"abstract":"Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available: https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .","url_abs":"https://arxiv.org/abs/2305.13009v3","url_pdf":"https://arxiv.org/pdf/2305.13009v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"textually-pretrained-speech-language-models","repo_url":"https://github.com/slp-rl/spokenstorycloze","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[{"slug":"sstorycloze","name":"sStoryCloze","full_name":"Spoken Story Cloze"},{"slug":"tstorycloze","name":"tStoryCloze","full_name":"Topic Story Cloze"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-salmon","task":"Language Modelling","dataset":"SALMon","model":"TWIST 7B","rank_in_archive_order":4,"of":10,"metrics":{"Background (Domain) Consistency":"55.0","Background (Random) Consistency":"60.5","Background Alignment":"54.5","Gender Consistency":"70.0","Room Consistency":"62.0","Sentiment Alignment":"51.5","Sentiment Consistency":"61.5","Speaker Consistency":"71.0"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-salmon","task":"Language Modelling","dataset":"SALMon","model":"TWIST 1.3B","rank_in_archive_order":5,"of":10,"metrics":{"Background (Domain) Consistency":"55.5","Background (Random) Consistency":"60.5","Background Alignment":"56.5","Gender Consistency":"69.5","Room Consistency":"59.0","Sentiment Alignment":"53.0","Sentiment Consistency":"61.5","Speaker Consistency":"69.0"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-salmon","task":"Language Modelling","dataset":"SALMon","model":"TWIST 350M","rank_in_archive_order":6,"of":10,"metrics":{"Background (Domain) Consistency":"54.0","Background (Random) Consistency":"61.5","Background Alignment":"56.5","Gender Consistency":"68.0","Room Consistency":"59.0","Sentiment Alignment":"51.5","Sentiment Consistency":"59.0","Speaker Consistency":"69.5"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.13009","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}