{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/syllablelm-learning-coarse-semantic-units-for","title":"SyllableLM: Learning Coarse Semantic Units for Speech Language Models","arxiv_id":"2410.04029","date":"2024-10-05","proceeding":null,"authors":["Alan Baade","Puyuan Peng","David Harwath"],"abstract":"Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, we introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with a novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, we successfully train SyllableLM, a Speech Language Model (SpeechLM) that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks. SyllableLM also achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup.","url_abs":"https://arxiv.org/abs/2410.04029v1","url_pdf":"https://arxiv.org/pdf/2410.04029v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"syllablelm-learning-coarse-semantic-units-for","repo_url":"https://github.com/alanbaade/SyllableLM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"speech-tokenization","task_name":"Speech Tokenization"},{"task_slug":"spoken-language-understanding","task_name":"Spoken Language Understanding"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.04029","atlas_url":"https://app.syntology.ai/?focus=2410.04029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alanbaade/SyllableLM","reach":null}],"summary":{"ran_fixture":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7a1b373c970a8042","entry":"efficient_extraction","repo":"alanbaade/SyllableLM","repo_kind":"official","path":"extract_units.py","file_url":"https://github.com/alanbaade/SyllableLM/blob/HEAD/extract_units.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a1b373c970a8042"}},{"code_sha256_prefix":"a6eacf5829b12d84","entry":"efficient_extraction_dp_helper","repo":"alanbaade/SyllableLM","repo_kind":"official","path":"extract_units.py","file_url":"https://github.com/alanbaade/SyllableLM/blob/HEAD/extract_units.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6eacf5829b12d84"}},{"code_sha256_prefix":"8186c17041c57836","entry":"get_quantile_borders_helper","repo":"alanbaade/SyllableLM","repo_kind":"official","path":"extract_units.py","file_url":"https://github.com/alanbaade/SyllableLM/blob/HEAD/extract_units.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8186c17041c57836"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}