{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-source-music-generation-with-latent","title":"Multi-Source Music Generation with Latent Diffusion","arxiv_id":"2409.06190","date":"2024-09-10","proceeding":null,"authors":["Zhongweiyang Xu","Debottam Dutta","Yu-Lin Wei","Romit Roy Choudhury"],"abstract":"Most music generation models directly generate a single music mixture. To allow for more flexible and controllable generation, the Multi-Source Diffusion Model (MSDM) has been proposed to model music as a mixture of multiple instrumental sources (e.g. piano, drums, bass, and guitar). Its goal is to use one single diffusion model to generate mutually-coherent music sources, that are then mixed to form the music. Despite its capabilities, MSDM is unable to generate music with rich melodies and often generates empty sounds. Its waveform diffusion approach also introduces significant Gaussian noise artifacts that compromise audio quality. In response, we introduce a Multi-Source Latent Diffusion Model (MSLDM) that employs Variational Autoencoders (VAEs) to encode each instrumental source into a distinct latent representation. By training a VAE on all music sources, we efficiently capture each source's unique characteristics in a \"source latent.\" The source latents are concatenated and our diffusion model learns this joint latent space. This approach significantly enhances the total and partial generation of music by leveraging the VAE's latent compression and noise-robustness. The compressed source latent also facilitates more efficient generation. Subjective listening tests and Frechet Audio Distance (FAD) scores confirm that our model outperforms MSDM, showcasing its practical and enhanced applicability in music generation systems. We also emphasize that modeling sources is more effective than direct music mixture modeling. Codes and models are available at https://github.com/XZWY/MSLDM. Demos are available at https://xzwy.github.io/MSLDMDemo/.","url_abs":"https://arxiv.org/abs/2409.06190v4","url_pdf":"https://arxiv.org/pdf/2409.06190v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-source-music-generation-with-latent","repo_url":"https://github.com/xzwy/msldm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"fad","task_name":"FAD"},{"task_slug":"music-generation","task_name":"Music Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.06190","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.06190"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xzwy/msldm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_fixture":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab52a777f7ea129a","entry":"get_logger","repo":"xzwy/msldm","repo_kind":"official","path":"msldm/main/utils.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/msldm/main/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab52a777f7ea129a"}},{"code_sha256_prefix":"17eb0a31c91b671c","entry":"normal_kl","repo":"xzwy/msldm","repo_kind":"official","path":"SourceVAE/modules/distributions.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/SourceVAE/modules/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17eb0a31c91b671c"}},{"code_sha256_prefix":"1714b81de8dbea7f","entry":"pad_dims","repo":"xzwy/msldm","repo_kind":"official","path":"msldm/main/diffusion.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/msldm/main/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1714b81de8dbea7f"}},{"code_sha256_prefix":"717f09b5b16b8d23","entry":"retry_if_error","repo":"xzwy/msldm","repo_kind":"official","path":"msldm/main/utils.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/msldm/main/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"717f09b5b16b8d23"}},{"code_sha256_prefix":"203f9e6f7f2d5dcb","entry":"sequential_mask","repo":"xzwy/msldm","repo_kind":"official","path":"msldm/main/diffusion.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/msldm/main/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"203f9e6f7f2d5dcb"}},{"code_sha256_prefix":"f247e4748f379a76","entry":"snake","repo":"xzwy/msldm","repo_kind":"official","path":"SourceVAE/modules/layers.py","file_url":"https://github.com/xzwy/msldm/blob/HEAD/SourceVAE/modules/layers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f247e4748f379a76"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}