{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/soundctm-uniting-score-based-and-consistency","title":"SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation","arxiv_id":"2405.18503","date":"2024-05-28","proceeding":null,"authors":["Koichi Saito","Dongjun Kim","Takashi Shibuya","Chieh-Hsin Lai","Zhi Zhong","Yuhta Takida","Yuki Mitsufuji"],"abstract":"Sound content creation, essential for multimedia works such as video games and films, often involves extensive trial-and-error, enabling creators to semantically reflect their artistic ideas and inspirations, which evolve throughout the creation process, into the sound. Recent high-quality diffusion-based Text-to-Sound (T2S) generative models provide valuable tools for creators. However, these models often suffer from slow inference speeds, imposing an undesirable burden that hinders the trial-and-error process. While existing T2S distillation models address this limitation through 1-step generation, the sample quality of $1$-step generation remains insufficient for production use. Additionally, while multi-step sampling in those distillation models improves sample quality itself, the semantic content changes due to their lack of deterministic sampling capabilities. To address these issues, we introduce Sound Consistency Trajectory Models (SoundCTM), which allow flexible transitions between high-quality $1$-step sound generation and superior sound quality through multi-step deterministic sampling. This allows creators to efficiently conduct trial-and-error with 1-step generation to semantically align samples with their intention, and subsequently refine sample quality with preserving semantic content through deterministic multi-step sampling. To develop SoundCTM, we reframe the CTM training framework, originally proposed in computer vision, and introduce a novel feature distance using the teacher network for a distillation loss. For production-level generation, we scale up our model to 1B trainable parameters, making SoundCTM-DiT-1B the first large-scale distillation model in the sound community to achieve both promising high-quality 1-step and multi-step full-band (44.1kHz) generation.","url_abs":"https://arxiv.org/abs/2405.18503v3","url_pdf":"https://arxiv.org/pdf/2405.18503v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"soundctm-uniting-score-based-and-consistency","repo_url":"https://github.com/sony/soundctm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":null,"task_name":"AudioCaps"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.18503","atlas_url":"https://app.syntology.ai/?focus=2405.18503","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18503"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sony/soundctm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30befb7e4327e615","entry":"append_dims","repo":"sony/soundctm","repo_kind":"official","path":"tango_edm/models_edm.py","file_url":"https://github.com/sony/soundctm/blob/HEAD/tango_edm/models_edm.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30befb7e4327e615"}},{"code_sha256_prefix":"6ce126db645cc0a7","entry":"load_state_dict","repo":"sony/soundctm","repo_kind":"official","path":"clap/clap_module/factory.py","file_url":"https://github.com/sony/soundctm/blob/HEAD/clap/clap_module/factory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ce126db645cc0a7"}},{"code_sha256_prefix":"3ab23e5d9f9352a8","entry":"create_model","repo":"sony/soundctm","repo_kind":"official","path":"clap/clap_module/factory.py","file_url":"https://github.com/sony/soundctm/blob/HEAD/clap/clap_module/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ab23e5d9f9352a8"}},{"code_sha256_prefix":"bfdd273dab42b860","entry":"create_model_and_transforms","repo":"sony/soundctm","repo_kind":"official","path":"clap/clap_module/factory.py","file_url":"https://github.com/sony/soundctm/blob/HEAD/clap/clap_module/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfdd273dab42b860"}},{"code_sha256_prefix":"8b16b0674af254db","entry":"edm_sampler","repo":"sony/soundctm","repo_kind":"official","path":"tango_edm/models_edm.py","file_url":"https://github.com/sony/soundctm/blob/HEAD/tango_edm/models_edm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b16b0674af254db"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}