{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generative-pre-trained-speech-language-model","title":"Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer","arxiv_id":"2406.00976","date":"2024-06-03","proceeding":null,"authors":["Yongxin Zhu","Dan Su","Liqiang He","Linli Xu","Dong Yu"],"abstract":"While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce \\textbf{G}enerative \\textbf{P}re-trained \\textbf{S}peech \\textbf{T}ransformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. The code is available at \\url{https://github.com/youngsheen/GPST}.","url_abs":"https://arxiv.org/abs/2406.00976v2","url_pdf":"https://arxiv.org/pdf/2406.00976v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generative-pre-trained-speech-language-model","repo_url":"https://github.com/youngsheen/gpst","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.00976","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.00976"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/youngsheen/gpst","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ccda3c47a635f4ba","entry":"custom_collate_fn","repo":"youngsheen/gpst","repo_kind":"official","path":"preprocess/data_handler.py","file_url":"https://github.com/youngsheen/gpst/blob/HEAD/preprocess/data_handler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccda3c47a635f4ba"}},{"code_sha256_prefix":"14adb6fc99cad5d8","entry":"init_distributed_context","repo":"youngsheen/gpst","repo_kind":"official","path":"preprocess/distributed.py","file_url":"https://github.com/youngsheen/gpst/blob/HEAD/preprocess/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14adb6fc99cad5d8"}},{"code_sha256_prefix":"7701aac4ae2dd9f6","entry":"random_crop_match","repo":"youngsheen/gpst","repo_kind":"official","path":"fairseq_user/GPST/dataset_lmdb.py","file_url":"https://github.com/youngsheen/gpst/blob/HEAD/fairseq_user/GPST/dataset_lmdb.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7701aac4ae2dd9f6"}},{"code_sha256_prefix":"dab9495277b7f3a3","entry":"worker_shard_path","repo":"youngsheen/gpst","repo_kind":"official","path":"preprocess/transcribe.py","file_url":"https://github.com/youngsheen/gpst/blob/HEAD/preprocess/transcribe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dab9495277b7f3a3"}},{"code_sha256_prefix":"a1d2db794ee5839b","entry":"extract_features_with_embedding","repo":"youngsheen/gpst","repo_kind":"official","path":"fairseq_user/GPST/util.py","file_url":"https://github.com/youngsheen/gpst/blob/HEAD/fairseq_user/GPST/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1d2db794ee5839b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}