{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/blsp-bootstrapping-language-speech-pre-1","title":"BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing","arxiv_id":"2309.00916","date":"2023-09-02","proceeding":null,"authors":["Chen Wang","Minpeng Liao","Zhongqiang Huang","Jinliang Lu","Junhong Wu","Yuchen Liu","Chengqing Zong","Jiajun Zhang"],"abstract":"The emergence of large language models (LLMs) has sparked significant interest in extending their remarkable language capabilities to speech. However, modality alignment between speech and text still remains an open problem. Current solutions can be categorized into two strategies. One is a cascaded approach where outputs (tokens or states) of a separately trained speech recognition system are used as inputs for LLMs, which limits their potential in modeling alignment between speech and text. The other is an end-to-end approach that relies on speech instruction data, which is very difficult to collect in large quantities. In this paper, we address these issues and propose the BLSP approach that Bootstraps Language-Speech Pre-training via behavior alignment of continuation writing. We achieve this by learning a lightweight modality adapter between a frozen speech encoder and an LLM, ensuring that the LLM exhibits the same generation behavior regardless of the modality of input: a speech segment or its transcript. The training process can be divided into two steps. The first step prompts an LLM to generate texts with speech transcripts as prefixes, obtaining text continuations. In the second step, these continuations are used as supervised signals to train the modality adapter in an end-to-end manner. We demonstrate that this straightforward process can extend the capabilities of LLMs to speech, enabling speech recognition, speech translation, spoken language understanding, and speech conversation, even in zero-shot cross-lingual scenarios.","url_abs":"https://arxiv.org/abs/2309.00916v2","url_pdf":"https://arxiv.org/pdf/2309.00916v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"blsp-bootstrapping-language-speech-pre-1","repo_url":"https://github.com/cwang621/blsp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"spoken-language-understanding","task_name":"Spoken Language Understanding"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.00916","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.00916"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cwang621/blsp","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"11be103f3c940844","entry":"collate_tokens","repo":"cwang621/blsp","repo_kind":"official","path":"blsp/src/speech_text_paired_dataset.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/blsp/src/speech_text_paired_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11be103f3c940844"}},{"code_sha256_prefix":"25e0884d6d058e54","entry":"collate_tokens","repo":"cwang621/blsp","repo_kind":"official","path":"data_process/asr_text_generation.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/data_process/asr_text_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25e0884d6d058e54"}},{"code_sha256_prefix":"115dd5cd6546803f","entry":"get_dataset","repo":"cwang621/blsp","repo_kind":"official","path":"data_process/asr_text_generation.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/data_process/asr_text_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"115dd5cd6546803f"}},{"code_sha256_prefix":"9bc4ef43619c4b45","entry":"get_shard_range","repo":"cwang621/blsp","repo_kind":"official","path":"data_process/asr_text_generation.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/data_process/asr_text_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc4ef43619c4b45"}},{"code_sha256_prefix":"74f6d53ff93d8db0","entry":"lengths_to_padding_mask","repo":"cwang621/blsp","repo_kind":"official","path":"blsp/src/modeling_blsp.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/blsp/src/modeling_blsp.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74f6d53ff93d8db0"}},{"code_sha256_prefix":"b23800765b9ccdff","entry":"load_text_instruction_dataset","repo":"cwang621/blsp","repo_kind":"official","path":"blsp/src/text_instruction_dataset.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/blsp/src/text_instruction_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b23800765b9ccdff"}},{"code_sha256_prefix":"e35ba6d4c2745c08","entry":"process_dataset","repo":"cwang621/blsp","repo_kind":"official","path":"blsp/src/text_instruction_dataset.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/blsp/src/text_instruction_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e35ba6d4c2745c08"}},{"code_sha256_prefix":"61134cad531c377a","entry":"clean_text","repo":"cwang621/blsp","repo_kind":"official","path":"data_process/prepare_alpaca.py","file_url":"https://github.com/cwang621/blsp/blob/HEAD/data_process/prepare_alpaca.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61134cad531c377a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}