{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mustango-toward-controllable-text-to-music","title":"Mustango: Toward Controllable Text-to-Music Generation","arxiv_id":"2311.08355","date":"2023-11-14","proceeding":null,"authors":["Jan Melechovsky","Zixun Guo","Deepanway Ghosal","Navonil Majumder","Dorien Herremans","Soujanya Poria"],"abstract":"The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.","url_abs":"https://arxiv.org/abs/2311.08355v3","url_pdf":"https://arxiv.org/pdf/2311.08355v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mustango-toward-controllable-text-to-music","repo_url":"https://github.com/amaai-lab/mustango","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mustango-toward-controllable-text-to-music","repo_url":"https://github.com/atharva20038/music4all","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}},{"paper_slug":"mustango-toward-controllable-text-to-music","repo_url":"https://github.com/MindSpore-scientific/code-10/tree/main/text-to-music","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"mustango-toward-controllable-text-to-music","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/2/musicgen_melody","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"music-generation","task_name":"Music Generation"},{"task_slug":"music-information-retrieval","task_name":"Music Information Retrieval"},{"task_slug":"text-to-music-generation","task_name":"Text-to-Music Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[{"slug":"musicbench","name":"MusicBench","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-music-generation-on-musicbench","task":"Text-to-Music Generation","dataset":"MusicBench","model":"Mustango (non-pretrained)","rank_in_archive_order":1,"of":1,"metrics":{"FAD":"2.09"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.08355","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.08355"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amaai-lab/mustango","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-10/tree/main/text-to-music","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-9/tree/main/2/musicgen_melody","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/atharva20038/music4all","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_honours":3,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"44ee71ec7cb0bca2","entry":"build_relative_position","repo":"amaai-lab/mustango","repo_kind":"official","path":"modelling_deberta_v2.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/modelling_deberta_v2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44ee71ec7cb0bca2"}},{"code_sha256_prefix":"dab32848a86bc88f","entry":"configure_dataloaders","repo":"amaai-lab/mustango","repo_kind":"official","path":"beats_model/train_beats.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/beats_model/train_beats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dab32848a86bc88f"}},{"code_sha256_prefix":"4dab6811065fbe1c","entry":"configure_optimizer","repo":"amaai-lab/mustango","repo_kind":"official","path":"beats_model/train_beats.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/beats_model/train_beats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dab6811065fbe1c"}},{"code_sha256_prefix":"a26f85d7c72ef39a","entry":"get_padding","repo":"amaai-lab/mustango","repo_kind":"official","path":"audioldm/hifigan/models.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/audioldm/hifigan/models.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a26f85d7c72ef39a"}},{"code_sha256_prefix":"0fc5d9cde41807d8","entry":"l2_norm","repo":"amaai-lab/mustango","repo_kind":"official","path":"layers/layers.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/layers/layers.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fc5d9cde41807d8"}},{"code_sha256_prefix":"cde5a0a5f4074d68","entry":"make_log_bucket_position","repo":"amaai-lab/mustango","repo_kind":"official","path":"modelling_deberta_v2.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/modelling_deberta_v2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cde5a0a5f4074d68"}},{"code_sha256_prefix":"e561180dff3faff6","entry":"rounding","repo":"amaai-lab/mustango","repo_kind":"official","path":"layers/layers.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/layers/layers.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e561180dff3faff6"}},{"code_sha256_prefix":"e24838aedd2adfc0","entry":"configure_scheduler","repo":"amaai-lab/mustango","repo_kind":"official","path":"beats_model/train_beats.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/beats_model/train_beats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e24838aedd2adfc0"}},{"code_sha256_prefix":"11c7e96cb235d1f7","entry":"get_mask","repo":"amaai-lab/mustango","repo_kind":"official","path":"modelling_deberta_v2.py","file_url":"https://github.com/amaai-lab/mustango/blob/HEAD/modelling_deberta_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11c7e96cb235d1f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}