{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/audioldm-2-learning-holistic-audio-generation","title":"AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining","arxiv_id":"2308.05734","date":"2023-08-10","proceeding":null,"authors":["Haohe Liu","Yi Yuan","Xubo Liu","Xinhao Mei","Qiuqiang Kong","Qiao Tian","Yuping Wang","Wenwu Wang","Yuxuan Wang","Mark D. Plumbley"],"abstract":"Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful consideration of specific objectives and biases that can significantly differ from those of other types. To bring us closer to a unified perspective of audio generation, this paper proposes a framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework introduces a general representation of audio, called \"language of audio\" (LOA). Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model. In the generation process, we translate any modalities into LOA by using a GPT-2 model, and we perform self-supervised audio generation learning with a latent diffusion model conditioned on LOA. The proposed framework naturally brings advantages such as in-context learning abilities and reusable self-supervised pretrained AudioMAE and latent diffusion models. Experiments on the major benchmarks of text-to-audio, text-to-music, and text-to-speech demonstrate state-of-the-art or competitive performance against previous approaches. Our code, pretrained model, and demo are available at https://audioldm.github.io/audioldm2.","url_abs":"https://arxiv.org/abs/2308.05734v3","url_pdf":"https://arxiv.org/pdf/2308.05734v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"audioldm-2-learning-holistic-audio-generation","repo_url":"https://github.com/haoheliu/AudioLDM2","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"audioldm-2-learning-holistic-audio-generation","repo_url":"https://github.com/haoheliu/audioldm-training-finetuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"text-to-music-generation","task_name":"Text-to-Music Generation"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-generation-on-audiocaps","task":"Audio Generation","dataset":"AudioCaps","model":"AudioLDM2-large","rank_in_archive_order":8,"of":23,"metrics":{"CLAP_LAION":"0.53","CLAP_MS":"0.37","FAD":"2.02","FD":"26.18","FD_openl3":"158.04","IS":"8.55","KL_passt":"1.68"},"uses_additional_data":false},{"leaderboard":"/sota/audio-generation-on-audiocaps","task":"Audio Generation","dataset":"AudioCaps","model":"AudioLDM 2-AC-Large","rank_in_archive_order":13,"of":23,"metrics":{"CLAP_LAION":"0.243","FAD":"1.42"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-music-generation-on-musiccaps","task":"Text-to-Music Generation","dataset":"MusicCaps","model":"AudioLDM2-large","rank_in_archive_order":8,"of":21,"metrics":{"CLAP_LAION":"0.48","CLAP_MS":"0.47","FAD":"2.93","FD":"16.34","FD_openl3":"190.16","IS":"2.59","KL_passt":"1.00"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-music-generation-on-musiccaps","task":"Text-to-Music Generation","dataset":"MusicCaps","model":"AudioLDM 2-Full","rank_in_archive_order":9,"of":21,"metrics":{"FAD":"3.13","KL_passt":"1.20"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-music-generation-on-musiccaps","task":"Text-to-Music Generation","dataset":"MusicCaps","model":"AudioLDM2-music","rank_in_archive_order":21,"of":21,"metrics":{"FD_openl3":"354.05","KL_passt":"1.53"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.05734","atlas_url":"https://app.syntology.ai/?focus=2308.05734","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.05734"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoheliu/AudioLDM2","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoheliu/audioldm-training-finetuning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_honours":2,"ran_violates":3,"unverified":14},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1},"listed":{"samples":14,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"22310b73059fa8cb","entry":"calculate_relative_bandwidth","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/dataset_plugin.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/dataset_plugin.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22310b73059fa8cb"}},{"code_sha256_prefix":"33e4f7a8bf80ee3a","entry":"get_bit_depth","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utils.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33e4f7a8bf80ee3a"}},{"code_sha256_prefix":"159fc130dc64d332","entry":"get_duration","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utils.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"159fc130dc64d332"}},{"code_sha256_prefix":"8f1fbba0d901fc52","entry":"get_map","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/clap/open_clip/loss.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/clap/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f1fbba0d901fc52"}},{"code_sha256_prefix":"a26f85d7c72ef39a","entry":"get_padding","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/hifigan/models.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/hifigan/models.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a26f85d7c72ef39a"}},{"code_sha256_prefix":"f1cba67eb6d6e4ad","entry":"get_param_num","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utilities/model.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utilities/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1cba67eb6d6e4ad"}},{"code_sha256_prefix":"7c29b374198bddcd","entry":"is_global_master","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/distributed.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c29b374198bddcd"}},{"code_sha256_prefix":"0377e309e95cfc3e","entry":"is_local_master","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/distributed.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0377e309e95cfc3e"}},{"code_sha256_prefix":"b092791002b40e8a","entry":"is_master","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/distributed.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b092791002b40e8a"}},{"code_sha256_prefix":"c7e729e9f89167e4","entry":"is_pretrained_params","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/lp_main.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/lp_main.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7e729e9f89167e4"}},{"code_sha256_prefix":"b18f6b953126d4e6","entry":"read_list","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utils.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b18f6b953126d4e6"}},{"code_sha256_prefix":"360696965b9c6f2a","entry":"set_cond_text","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/pipeline.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"360696965b9c6f2a"}},{"code_sha256_prefix":"4dab43e0c6c8c170","entry":"torch_version_orig_mod_remove","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utilities/model.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utilities/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4dab43e0c6c8c170"}},{"code_sha256_prefix":"aafcff3ad1ccef6e","entry":"accuracy","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/zero_shot.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/zero_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aafcff3ad1ccef6e"}},{"code_sha256_prefix":"fde7529206b8c46f","entry":"build_model_from_openai_state_dict","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/clap/open_clip/model.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/clap/open_clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fde7529206b8c46f"}},{"code_sha256_prefix":"932d44bbf8bd0e3f","entry":"cosine_lr","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/scheduler.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"932d44bbf8bd0e3f"}},{"code_sha256_prefix":"9b31c784540df0d7","entry":"gather_features","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/clap/open_clip/loss.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/clap/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b31c784540df0d7"}},{"code_sha256_prefix":"89bb9be7815a9834","entry":"get_available_checkpoint_keys","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/utilities/model.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/utilities/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89bb9be7815a9834"}},{"code_sha256_prefix":"575ba28efe232215","entry":"get_default_params","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/params.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"575ba28efe232215"}},{"code_sha256_prefix":"0186ae56bb268a36","entry":"get_vits_phoneme_ids","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/dataset_plugin.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/dataset_plugin.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0186ae56bb268a36"}},{"code_sha256_prefix":"ad984e4532566626","entry":"get_vits_phoneme_ids_no_padding","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/dataset_plugin.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/dataset_plugin.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad984e4532566626"}},{"code_sha256_prefix":"47bbdd629908bce3","entry":"int16_to_float32","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/data.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47bbdd629908bce3"}},{"code_sha256_prefix":"473061649d09d30c","entry":"lp_gather_features","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/clap/open_clip/loss.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/clap/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"473061649d09d30c"}},{"code_sha256_prefix":"7ffcc636c121640f","entry":"run","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/zero_shot.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/zero_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ffcc636c121640f"}},{"code_sha256_prefix":"6c018f9e8c31090c","entry":"trace_model","repo":"haoheliu/AudioLDM2","repo_kind":"official","path":"audioldm2/clap/open_clip/model.py","file_url":"https://github.com/haoheliu/AudioLDM2/blob/HEAD/audioldm2/clap/open_clip/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c018f9e8c31090c"}},{"code_sha256_prefix":"2bf62c0287050bdd","entry":"unwrap_model","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/lp_train.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/lp_train.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bf62c0287050bdd"}},{"code_sha256_prefix":"4ccd5f0108a2c3f2","entry":"update_top_k_performance","repo":"haoheliu/audioldm-training-finetuning","repo_kind":"listed","path":"audioldm_train/modules/clap/training/lp_main.py","file_url":"https://github.com/haoheliu/audioldm-training-finetuning/blob/HEAD/audioldm_train/modules/clap/training/lp_main.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ccd5f0108a2c3f2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}