{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/muq-self-supervised-music-representation","title":"MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization","arxiv_id":"2501.01108","date":"2025-01-02","proceeding":null,"authors":["Haina Zhu","Yizhi Zhou","Hangting Chen","Jianwei Yu","Ziyang Ma","Rongzhi Gu","Yi Luo","Wei Tan","Xie Chen"],"abstract":"Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detection, and more. In this paper, we propose a self-supervised music representation learning model for music understanding. Distinguished from previous studies adopting random projection or existing neural codec, the proposed model, named MuQ, is trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ). Our Mel-RVQ utilizes residual linear projection structure for Mel spectrum quantization to enhance the stability and efficiency of target extraction and lead to better performance. Experiments in a large variety of downstream tasks demonstrate that MuQ outperforms previous self-supervised music representation models with only 0.9K hours of open-source pre-training data. Scaling up the data to over 160K hours and adopting iterative training consistently improve the model performance. To further validate the strength of our model, we present MuQ-MuLan, a joint music-text embedding model based on contrastive learning, which achieves state-of-the-art performance in the zero-shot music tagging task on the MagnaTagATune dataset. Code and checkpoints are open source in https://github.com/tencent-ailab/MuQ.","url_abs":"https://arxiv.org/abs/2501.01108v2","url_pdf":"https://arxiv.org/pdf/2501.01108v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"muq-self-supervised-music-representation","repo_url":"https://github.com/tencent-ailab/muq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"key-detection","task_name":"Key Detection"},{"task_slug":"music-tagging","task_name":"Music Tagging"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2501.01108","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.01108"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencent-ailab/muq","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"feb6de6bd3620143","entry":"pair","repo":"tencent-ailab/muq","repo_kind":"official","path":"src/muq/muq_mulan/models/audio.py","file_url":"https://github.com/tencent-ailab/muq/blob/HEAD/src/muq/muq_mulan/models/audio.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"feb6de6bd3620143"}},{"code_sha256_prefix":"9effe43d176f8205","entry":"interspersed_indices","repo":"tencent-ailab/muq","repo_kind":"official","path":"src/muq/muq_mulan/modules/contrastive.py","file_url":"https://github.com/tencent-ailab/muq/blob/HEAD/src/muq/muq_mulan/modules/contrastive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"9effe43d176f8205"}},{"code_sha256_prefix":"d52bf6e1f34bb253","entry":"matrix_diag","repo":"tencent-ailab/muq","repo_kind":"official","path":"src/muq/muq_mulan/modules/contrastive.py","file_url":"https://github.com/tencent-ailab/muq/blob/HEAD/src/muq/muq_mulan/modules/contrastive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d52bf6e1f34bb253"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}