{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hybrid-transformers-for-music-source","title":"Hybrid Transformers for Music Source Separation","arxiv_id":"2211.08553","date":"2022-11-15","proceeding":null,"authors":["Simon Rouard","Francisco Massa","Alexandre Défossez"],"abstract":"A natural question arising in Music Source Separation (MSS) is whether long range contextual information is useful, or whether local acoustic features are sufficient. In other fields, attention based Transformers have shown their ability to integrate information over long sequences. In this work, we introduce Hybrid Transformer Demucs (HT Demucs), an hybrid temporal/spectral bi-U-Net based on Hybrid Demucs, where the innermost layers are replaced by a cross-domain Transformer Encoder, using self-attention within one domain, and cross-attention across domains. While it performs poorly when trained only on MUSDB, we show that it outperforms Hybrid Demucs (trained on the same data) by 0.45 dB of SDR when using 800 extra training songs. Using sparse attention kernels to extend its receptive field, and per source fine-tuning, we achieve state-of-the-art results on MUSDB with extra training data, with 9.20 dB of SDR.","url_abs":"https://arxiv.org/abs/2211.08553v1","url_pdf":"https://arxiv.org/pdf/2211.08553v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hybrid-transformers-for-music-source","repo_url":"https://github.com/facebookresearch/demucs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hybrid-transformers-for-music-source","repo_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"music-source-separation","task_name":"Music Source Separation"},{"task_slug":"speech-enhancement","task_name":"Speech Enhancement"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/music-source-separation-on-musdb18","task":"Music Source Separation","dataset":"MUSDB18","model":"Sparse HT Demucs (fine tuned)","rank_in_archive_order":1,"of":27,"metrics":{"SDR (avg)":"9.20","SDR (bass)":"10.47","SDR (drums)":"10.83","SDR (other)":"6.41","SDR (vocals)":"9.37"},"uses_additional_data":true},{"leaderboard":"/sota/music-source-separation-on-musdb18","task":"Music Source Separation","dataset":"MUSDB18","model":"Hybrid Transformer Demucs (f.t.)","rank_in_archive_order":2,"of":27,"metrics":{"SDR (avg)":"9.00","SDR (bass)":"9.78","SDR (drums)":"10.08","SDR (other)":"6.42","SDR (vocals)":"9.20"},"uses_additional_data":true},{"leaderboard":"/sota/music-source-separation-on-musdb18-hq","task":"Music Source Separation","dataset":"MUSDB18-HQ","model":"Sparse HT Demucs (fine tuned)","rank_in_archive_order":4,"of":14,"metrics":{"SDR (avg)":"9.20","SDR (bass)":"10.47","SDR (drums)":"10.83","SDR (others)":"6.41","SDR (vocals)":"9.37"},"uses_additional_data":true},{"leaderboard":"/sota/music-source-separation-on-musdb18-hq","task":"Music Source Separation","dataset":"MUSDB18-HQ","model":"Hybrid Transformer Demucs (f.t.)","rank_in_archive_order":5,"of":14,"metrics":{"SDR (avg)":"9.00","SDR (bass)":"10.39","SDR (drums)":"10.08","SDR (others)":"6.32","SDR (vocals)":"9.20"},"uses_additional_data":true},{"leaderboard":"/sota/speech-enhancement-on-ears-wham","task":"Speech Enhancement","dataset":"EARS-WHAM","model":"Demucs v4","rank_in_archive_order":3,"of":6,"metrics":{"DNSMOS":"3.66","ESTOI":"0.71","PESQ-WB":"2.37","POLQA":"2.97","SI-SDR":"16.92","SIGMOS":"2.87"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.08553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.08553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/demucs","reach":null}],"summary":{"unverified":10},"by_repo_kind":{"listed":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"a68dbf50b2629a55","entry":"average_metric","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/utils.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a68dbf50b2629a55"}},{"code_sha256_prefix":"ffd24b3bb5a4d744","entry":"build_metadata","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/compressed.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/compressed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ffd24b3bb5a4d744"}},{"code_sha256_prefix":"f89970356c8e9506","entry":"center_trim","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/utils.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f89970356c8e9506"}},{"code_sha256_prefix":"c20ada1396105c22","entry":"chose_norm","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/tasnet.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/tasnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c20ada1396105c22"}},{"code_sha256_prefix":"ec2fa0636cd11df4","entry":"device_id_to_process_device_map","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/__main__.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/__main__.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ec2fa0636cd11df4"}},{"code_sha256_prefix":"07fa2ca4b8bf0830","entry":"downsample","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/model.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"07fa2ca4b8bf0830"}},{"code_sha256_prefix":"f747e6cf6fc620a6","entry":"free_port","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/utils.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f747e6cf6fc620a6"}},{"code_sha256_prefix":"dccae3e25e1c55a2","entry":"get_name","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/parser.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"dccae3e25e1c55a2"}},{"code_sha256_prefix":"6fc6cdb0729e7905","entry":"overlap_and_add","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/tasnet.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/tasnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"6fc6cdb0729e7905"}},{"code_sha256_prefix":"5cf51541da88d14a","entry":"upsample","repo":"zhaozhipeng1997/demucs_ascend910_pytorch","repo_kind":"listed","path":"demucs/model.py","file_url":"https://github.com/zhaozhipeng1997/demucs_ascend910_pytorch/blob/HEAD/demucs/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5cf51541da88d14a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}