{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uniaudio-1-5-large-language-model-driven","title":"UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner","arxiv_id":"2406.10056","date":"2024-06-14","proceeding":null,"authors":["Dongchao Yang","Haohan Guo","Yuanyuan Wang","Rongjie Huang","Xiang Li","Xu Tan","Xixin Wu","Helen Meng"],"abstract":"The Large Language models (LLMs) have demonstrated supreme capabilities in text understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes a cross-modal in-context learning approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. Specifically, we propose a novel and LLMs-driven audio codec model, LLM-Codec, to transfer the audio modality into the textual space, \\textit{i.e.} representing audio tokens with words or sub-words in the vocabulary of LLMs, while keeping high audio reconstruction quality. The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into a well-trained LLMs token space. Thus, the audio representation can be viewed as a new \\textit{foreign language}, and LLMs can learn the new \\textit{foreign language} with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, \\textit{e.g.} speech emotion classification, audio classification, text-to-speech generation, speech enhancement, etc. The experimental results demonstrate that the LLMs equipped with the proposed LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, can achieve the expected functions in simple scenarios. It validates the feasibility and effectiveness of the proposed cross-modal in-context learning approach. To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.","url_abs":"https://arxiv.org/abs/2406.10056v1","url_pdf":"https://arxiv.org/pdf/2406.10056v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"uniaudio-1-5-large-language-model-driven","repo_url":"https://github.com/yangdongchao/llm-codec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.10056","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10056"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangdongchao/llm-codec","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_honours":1,"ran_violates":1,"ran_fixture":2,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"6d3d810524eb100b","entry":"apply_rotary_pos_emb","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"codec/attention.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/codec/attention.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d3d810524eb100b"}},{"code_sha256_prefix":"a26f85d7c72ef39a","entry":"get_padding","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"codec/module.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/codec/module.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a26f85d7c72ef39a"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"llama_inference/llama/model.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/llama_inference/llama/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"llama_inference/llama/model.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/llama_inference/llama/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"1147383164fd3a66","entry":"rotate_half","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"codec/attention.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/codec/attention.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1147383164fd3a66"}},{"code_sha256_prefix":"f247e4748f379a76","entry":"snake","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"codec/layers.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/codec/layers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f247e4748f379a76"}},{"code_sha256_prefix":"7f04dbcdf72d68fd","entry":"apply_rotary_emb","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"llama_inference/llama/model.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/llama_inference/llama/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f04dbcdf72d68fd"}},{"code_sha256_prefix":"35ac4a3a2d0faeac","entry":"get_2d_padding","repo":"yangdongchao/llm-codec","repo_kind":"official","path":"codec/msstftd.py","file_url":"https://github.com/yangdongchao/llm-codec/blob/HEAD/codec/msstftd.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35ac4a3a2d0faeac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}