{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-11096","title":"Efficient Training for Cross-lingual Speech Language Models","arxiv_id":"2604.11096","date":"2026-04-13","proceeding":null,"authors":["Yan Zhou","Qingkai Fang","Yun Hong","Yang Feng"],"abstract":"Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)","url_abs":"https://arxiv.org/abs/2604.11096","url_pdf":"https://arxiv.org/pdf/2604.11096","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.11096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.11096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/deepspeedai/DeepSpeed","reach":null}],"summary":{"ran":1,"ran_honours":4,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4346fff083939110","entry":"AutoTPMeta","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4346fff083939110"}},{"code_sha256_prefix":"6838070708760fe4","entry":"_attention_head_count_from","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6838070708760fe4"}},{"code_sha256_prefix":"1fd65563e4f7ec38","entry":"_kv_head_count_from","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fd65563e4f7ec38"}},{"code_sha256_prefix":"52ac3e38db39cd2f","entry":"get_shard_size","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52ac3e38db39cd2f"}},{"code_sha256_prefix":"349bb5104a5aa798","entry":"get_shard_size_list","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"349bb5104a5aa798"}},{"code_sha256_prefix":"556e5b0b420e135d","entry":"uneven_heads_all2all","repo":"deepspeedai/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/sequence/layer.py","file_url":"https://github.com/deepspeedai/DeepSpeed/blob/HEAD/deepspeed/sequence/layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"556e5b0b420e135d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}