{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/m2d2-exploring-general-purpose-audio-language","title":"M2D2: Exploring General-purpose Audio-Language Representations Beyond CLAP","arxiv_id":"2503.22104","date":"2025-03-28","proceeding":null,"authors":["Daisuke Niizumi","Daiki Takeuchi","Masahiro Yasuda","Binh Thien Nguyen","Yasunori Ohishi","Noboru Harada"],"abstract":"Contrastive language-audio pre-training (CLAP) has addressed audio-language tasks such as audio-text retrieval by aligning audio and text in a common feature space. While CLAP addresses general audio-language tasks, its audio features do not generalize well in audio tasks. In contrast, self-supervised learning (SSL) models learn general-purpose audio features that perform well in diverse audio tasks. We pursue representation learning that can be widely used in audio applications and hypothesize that a method that learns both general audio features and CLAP features should achieve our goal, which we call a general-purpose audio-language representation. To implement our hypothesis, we propose M2D2, a second-generation masked modeling duo (M2D) that combines an SSL M2D and CLAP. M2D2 learns two types of features using two modalities (audio and text) in a two-stage training process. It also utilizes advanced LLM-based sentence embeddings in CLAP training for powerful semantic supervision. In the first stage, M2D2 learns generalizable audio features from M2D and CLAP, where CLAP aligns the features with the fine LLM-based semantic embeddings. In the second stage, it learns CLAP features using the audio features learned from the LLM-based embeddings. Through these pre-training stages, M2D2 should enhance generalizability and performance in its audio and CLAP features. Experiments validated that M2D2 achieves effective general-purpose audio-language representation, highlighted with SOTA fine-tuning mAP of 49.0 for AudioSet, SOTA performance in music tasks, and top-level performance in audio-language tasks.","url_abs":"https://arxiv.org/abs/2503.22104v1","url_pdf":"https://arxiv.org/pdf/2503.22104v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m2d2-exploring-general-purpose-audio-language","repo_url":"https://github.com/nttcslab/eval-audio-repr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"m2d2-exploring-general-purpose-audio-language","repo_url":"https://github.com/nttcslab/m2d","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"audio-tagging","task_name":"Audio Tagging"},{"task_slug":"audio-captioning","task_name":"Audio captioning"},{"task_slug":"audio-to-text-retrieval","task_name":"Audio to Text Retrieval"},{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"instrument-recognition","task_name":"Instrument Recognition"},{"task_slug":"music-auto-tagging","task_name":"Music Auto-Tagging"},{"task_slug":"music-classification","task_name":"Music Classification"},{"task_slug":"music-genre-classification","task_name":"Music Genre Classification"},{"task_slug":"music-tagging","task_name":"Music Tagging"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"sentence-embeddings","task_name":"Sentence Embeddings"},{"task_slug":"singer-identification","task_name":"Singer Identification"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"},{"task_slug":"vocal-technique-classification","task_name":"Vocal technique classification"}],"methods":[{"method_slug":"m2d","method_name":"M2D"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"M2D2","rank_in_archive_order":15,"of":51,"metrics":{"Test mAP":"0.490"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"M2D2 AS+","rank_in_archive_order":3,"of":29,"metrics":{"Accuracy (5-fold)":"98.5","PRE-TRAINING DATASET":"AudioSet,WavCaps","Top-1 Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D-CLAP","rank_in_archive_order":1,"of":5,"metrics":{"EmoA":"77.4","EmoV":"61.9"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D2","rank_in_archive_order":2,"of":5,"metrics":{"EmoA":"76.7","EmoV":"59.3"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D","rank_in_archive_order":3,"of":5,"metrics":{"EmoA":"76.1","EmoV":"59.4"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D-CLAP","rank_in_archive_order":1,"of":7,"metrics":{"Accuracy":"80.6"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D2 AS+","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"79.7"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D AS","rank_in_archive_order":3,"of":7,"metrics":{"Accuracy":"78.7"},"uses_additional_data":false},{"leaderboard":"/sota/music-auto-tagging-on-magnatagatune","task":"Music Auto-Tagging","dataset":"MagnaTagATune","model":"M2D2 AS+","rank_in_archive_order":1,"of":3,"metrics":{"PR-AUC":"41.6","ROC AUC":"91.8"},"uses_additional_data":false},{"leaderboard":"/sota/singer-identification-on-vocalset-1","task":"Singer Identification","dataset":"VocalSet","model":"M2D2 AS+","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"92.7"},"uses_additional_data":true},{"leaderboard":"/sota/singer-identification-on-vocalset-1","task":"Singer Identification","dataset":"VocalSet","model":"M2D2","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"91.8"},"uses_additional_data":true},{"leaderboard":"/sota/vocal-technique-classification-on-vocalset-1","task":"Vocal technique classification","dataset":"VocalSet","model":"M2D2 AS+","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy ":"78.9"},"uses_additional_data":true},{"leaderboard":"/sota/vocal-technique-classification-on-vocalset-1","task":"Vocal technique classification","dataset":"VocalSet","model":"M2D2","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy ":"77.4"},"uses_additional_data":true}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}