{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/wavjourney-compositional-audio-creation-with","title":"WavJourney: Compositional Audio Creation with Large Language Models","arxiv_id":"2307.14335","date":"2023-07-26","proceeding":null,"authors":["Xubo Liu","Zhongkai Zhu","Haohe Liu","Yi Yuan","Meng Cui","Qiushi Huang","Jinhua Liang","Yin Cao","Qiuqiang Kong","Mark D. Plumbley","Wenwu Wang"],"abstract":"Despite breakthroughs in audio generation models, their capabilities are often confined to domain-specific conditions such as speech transcriptions and audio captions. However, real-world audio creation aims to generate harmonious audio containing various elements such as speech, music, and sound effects with controllable conditions, which is challenging to address using existing audio generation systems. We present WavJourney, a novel framework that leverages Large Language Models (LLMs) to connect various audio models for audio creation. WavJourney allows users to create storytelling audio content with diverse audio elements simply from textual descriptions. Specifically, given a text instruction, WavJourney first prompts LLMs to generate an audio script that serves as a structured semantic representation of audio elements. The audio script is then converted into a computer program, where each line of the program calls a task-specific audio generation model or computational operation function. The computer program is then executed to obtain a compositional and interpretable solution for audio creation. Experimental results suggest that WavJourney is capable of synthesizing realistic audio aligned with textually-described semantic, spatial and temporal conditions, achieving state-of-the-art results on text-to-audio generation benchmarks. Additionally, we introduce a new multi-genre story benchmark. Subjective evaluations demonstrate the potential of WavJourney in crafting engaging storytelling audio content from text. We further demonstrate that WavJourney can facilitate human-machine co-creation in multi-round dialogues. To foster future research, the code and synthesized audio are available at: https://audio-agi.github.io/WavJourney_demopage/.","url_abs":"https://arxiv.org/abs/2307.14335v2","url_pdf":"https://arxiv.org/pdf/2307.14335v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"wavjourney-compositional-audio-creation-with","repo_url":"https://github.com/audio-agi/wavjourney","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.14335","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.14335"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/audio-agi/wavjourney","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"af242371f0df6717","entry":"chat_with_gpt","repo":"audio-agi/wavjourney","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"af242371f0df6717"}},{"code_sha256_prefix":"4dd30a7a78c5ac92","entry":"convert_char_voice_map_to_md","repo":"audio-agi/wavjourney","repo_kind":"official","path":"ui_client.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/ui_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4dd30a7a78c5ac92"}},{"code_sha256_prefix":"58a330b153b41aae","entry":"extract_substring_with_quotes","repo":"audio-agi/wavjourney","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"58a330b153b41aae"}},{"code_sha256_prefix":"0dba0c9a60770cf0","entry":"get_file_content","repo":"audio-agi/wavjourney","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0dba0c9a60770cf0"}},{"code_sha256_prefix":"b74a579d793510bf","entry":"get_session_audio_path","repo":"audio-agi/wavjourney","repo_kind":"official","path":"utils.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b74a579d793510bf"}},{"code_sha256_prefix":"7c720f74c1dd4162","entry":"get_session_path","repo":"audio-agi/wavjourney","repo_kind":"official","path":"utils.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7c720f74c1dd4162"}},{"code_sha256_prefix":"b7b27860746beecb","entry":"get_session_voice_preset_path","repo":"audio-agi/wavjourney","repo_kind":"official","path":"utils.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b7b27860746beecb"}},{"code_sha256_prefix":"ec27e2963015338a","entry":"collect_and_check_audio_data","repo":"audio-agi/wavjourney","repo_kind":"official","path":"code_generator.py","file_url":"https://github.com/audio-agi/wavjourney/blob/HEAD/code_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ec27e2963015338a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}