Papers › Textually Pretrained Speech Language Models
Textually Pretrained Speech Language Models
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, Yossi Adi
Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available: https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | SALMon | TWIST 7B | Background (Domain) Consistency | 55.0 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Background (Random) Consistency | 60.5 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Background Alignment | 54.5 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Gender Consistency | 70.0 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Room Consistency | 62.0 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Sentiment Alignment | 51.5 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Sentiment Consistency | 61.5 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 7B | Speaker Consistency | 71.0 | #4 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Background (Domain) Consistency | 55.5 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Background (Random) Consistency | 60.5 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Background Alignment | 56.5 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Gender Consistency | 69.5 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Room Consistency | 59.0 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Sentiment Alignment | 53.0 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Sentiment Consistency | 61.5 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 1.3B | Speaker Consistency | 69.0 | #5 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Background (Domain) Consistency | 54.0 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Background (Random) Consistency | 61.5 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Background Alignment | 56.5 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Gender Consistency | 68.0 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Room Consistency | 59.0 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Sentiment Alignment | 51.5 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Sentiment Consistency | 59.0 | #6 of 10 | Archive leaderboard | report |
| Language Modelling | SALMon | TWIST 350M | Speaker Consistency | 69.5 | #6 of 10 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections