{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/weakly-supervised-audio-separation-via-bi","title":"Weakly-supervised Audio Separation via Bi-modal Semantic Similarity","arxiv_id":"2404.01740","date":"2024-04-02","proceeding":null,"authors":["Tanvir Mahmud","Saeed Amizadeh","Kazuhito Koishida","Diana Marculescu"],"abstract":"Conditional sound separation in multi-source audio mixtures without having access to single source sound data during training is a long standing challenge. Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for single source separation cases during training. However, in the case of language-conditional audio separation, we do have access to corresponding text descriptions for each audio mixture in our training data, which can be seen as (rough) representations of the audio samples in the language modality. To this end, in this paper, we propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality (i.e., audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples in the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, we propose to incorporate our framework into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between training and test samples. In particular, we show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervised framework, that enables a powerful semi-supervised framework for audio separation.","url_abs":"https://arxiv.org/abs/2404.01740v1","url_pdf":"https://arxiv.org/pdf/2404.01740v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"weakly-supervised-audio-separation-via-bi","repo_url":"https://github.com/halvani/constituent-treelib","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"weakly-supervised-audio-separation-via-bi","repo_url":"https://github.com/microsoft/bimodalaudioseparation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.01740","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.01740"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/BiModalAudioSeparation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/bimodalaudioseparation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/halvani/constituent-treelib","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"84919ee649260509","entry":"BlockCA","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84919ee649260509"}},{"code_sha256_prefix":"b733542e13e4b8ce","entry":"BlockCat","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b733542e13e4b8ce"}},{"code_sha256_prefix":"8ec43e0a25ffe65f","entry":"BlockIdentity","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ec43e0a25ffe65f"}},{"code_sha256_prefix":"903f2610f1453cea","entry":"BlockSA","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"903f2610f1453cea"}},{"code_sha256_prefix":"837dd2dcaeaaded4","entry":"CrossAttention","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"837dd2dcaeaaded4"}},{"code_sha256_prefix":"033bf6b0228555c7","entry":"Mlp","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"033bf6b0228555c7"}},{"code_sha256_prefix":"a84641b3af546ce0","entry":"ResBlock","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a84641b3af546ce0"}},{"code_sha256_prefix":"fee297065b6d46a5","entry":"SelfAttention","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fee297065b6d46a5"}},{"code_sha256_prefix":"af72484d5cfd7097","entry":"load_state_dict","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"models/laion_clap/clap_module/factory.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/models/laion_clap/clap_module/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af72484d5cfd7097"}},{"code_sha256_prefix":"81e9a172efc0d908","entry":"realimag","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"utils.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81e9a172efc0d908"}},{"code_sha256_prefix":"a4a08206de81190b","entry":"warpgrid","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"utils.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4a08206de81190b"}},{"code_sha256_prefix":"da16db8294ed4818","entry":"BlockSCA","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da16db8294ed4818"}},{"code_sha256_prefix":"e4e711f2a5fb5898","entry":"BlockSCACat","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4e711f2a5fb5898"}},{"code_sha256_prefix":"9811209e675df9fc","entry":"CondAttUNet","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9811209e675df9fc"}},{"code_sha256_prefix":"bdd91cc0612d6448","entry":"CondAttUNetBlock","repo":"microsoft/bimodalaudioseparation","repo_kind":"official","path":"models/cond_unet_attn.py","file_url":"https://github.com/microsoft/bimodalaudioseparation/blob/HEAD/models/cond_unet_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd91cc0612d6448"}},{"code_sha256_prefix":"5c0daaa41ed490ea","entry":"create_model","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"models/laion_clap/clap_module/factory.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/models/laion_clap/clap_module/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0daaa41ed490ea"}},{"code_sha256_prefix":"ffdbb019791354d1","entry":"create_model_and_transforms","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"models/laion_clap/clap_module/factory.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/models/laion_clap/clap_module/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffdbb019791354d1"}},{"code_sha256_prefix":"3e5b3b6994917c24","entry":"recover_rgb","repo":"microsoft/BiModalAudioSeparation","repo_kind":"official","path":"utils.py","file_url":"https://github.com/microsoft/BiModalAudioSeparation/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e5b3b6994917c24"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}