{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmsu-a-massive-multi-task-spoken-language","title":"MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark","arxiv_id":"2506.04779","date":"2025-06-05","proceeding":null,"authors":["Dingdong Wang","Jincenzi Wu","Junan Li","Dongchao Yang","Xueyuan Chen","Tianhua Zhang","Helen Meng"],"abstract":"Speech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken language understanding, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio information, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in spoken language. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. To ground our benchmark in linguistic theory, we systematically incorporate a wide range of linguistic phenomena, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics. Through a rigorous evaluation of 14 advanced SpeechLLMs, we identify substantial room for improvement in existing models, highlighting meaningful directions for future optimization. MMSU establishes a new standard for comprehensive assessment of spoken language understanding, providing valuable insights for developing more sophisticated human-AI speech interaction systems. MMSU benchmark is available at https://huggingface.co/datasets/ddwang2000/MMSU. Evaluation Code is available at https://github.com/dingdongwang/MMSU_Bench.","url_abs":"https://arxiv.org/abs/2506.04779v1","url_pdf":"https://arxiv.org/pdf/2506.04779v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmsu-a-massive-multi-task-spoken-language","repo_url":"https://github.com/dingdongwang/mmsu_bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}},{"paper_slug":"mmsu-a-massive-multi-task-spoken-language","repo_url":"https://github.com/QwenLM/Qwen2.5-Omni","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"rhythm","task_name":"Rhythm"},{"task_slug":"spoken-language-understanding","task_name":"Spoken Language Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.04779","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.04779"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/QwenLM/Qwen2.5-Omni","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dingdongwang/mmsu_bench","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}}],"summary":{"ran_honours":3,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"listed":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"6e45201fa27cb24a","entry":"ceil_by_factor","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6e45201fa27cb24a"}},{"code_sha256_prefix":"8155263d7ff19bb3","entry":"floor_by_factor","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8155263d7ff19bb3"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"e252767324188623","entry":"round_by_factor","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/qwen-omni-utils/src/qwen_omni_utils/v2_5/vision_process.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e252767324188623"}},{"code_sha256_prefix":"1a7a4178e97668c1","entry":"apply_multimodal_rotary_pos_emb","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1a7a4178e97668c1"}},{"code_sha256_prefix":"c4adc11c6d565fb8","entry":"apply_rotary_pos_emb_vision","repo":"QwenLM/Qwen2.5-Omni","repo_kind":"listed","path":"low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","file_url":"https://github.com/QwenLM/Qwen2.5-Omni/blob/HEAD/low-VRAM-mode/modeling_qwen2_5_omni_low_VRAM_mode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c4adc11c6d565fb8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}