{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/masked-modeling-duo-towards-a-universal-audio","title":"Masked Modeling Duo: Towards a Universal Audio Pre-training Framework","arxiv_id":"2404.06095","date":"2024-04-09","proceeding":null,"authors":["Daisuke Niizumi","Daiki Takeuchi","Yasunori Ohishi","Noboru Harada","Kunio Kashino"],"abstract":"Self-supervised learning (SSL) using masked prediction has made great strides in general-purpose audio representation. This study proposes Masked Modeling Duo (M2D), an improved masked prediction SSL, which learns by predicting representations of masked input signals that serve as training signals. Unlike conventional methods, M2D obtains a training signal by encoding only the masked part, encouraging the two networks in M2D to model the input. While M2D improves general-purpose audio representations, a specialized representation is essential for real-world applications, such as in industrial and medical domains. The often confidential and proprietary data in such domains is typically limited in size and has a different distribution from that in pre-training datasets. Therefore, we propose M2D for X (M2D-X), which extends M2D to enable the pre-training of specialized representations for an application X. M2D-X learns from M2D and an additional task and inputs background noise. We make the additional task configurable to serve diverse applications, while the background noise helps learn on small data and forms a denoising task that makes representation robust. With these design choices, M2D-X should learn a representation specialized to serve various application needs. Our experiments confirmed that the representations for general-purpose audio, specialized for the highly competitive AudioSet and speech domain, and a small-data medical task achieve top-level performance, demonstrating the potential of using our models as a universal audio pre-training framework. Our code is available online for future studies at https://github.com/nttcslab/m2d","url_abs":"https://arxiv.org/abs/2404.06095v1","url_pdf":"https://arxiv.org/pdf/2404.06095v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"masked-modeling-duo-towards-a-universal-audio","repo_url":"https://github.com/nttcslab/m2d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"masked-modeling-duo-towards-a-universal-audio","repo_url":"https://github.com/nttcslab/eval-audio-repr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"environment-sound-classification","task_name":"Environment Sound Classification"},{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speaker-identification","task_name":"Speaker Identification"}],"methods":[{"method_slug":"byol","method_name":"BYOL"},{"method_slug":"m2d","method_name":"M2D"}],"datasets_introduced":[],"methods_introduced":[{"slug":"m2d","name":"M2D","full_name":"Masked Modeling Duo"}],"results":[{"leaderboard":"/sota/on-google-speech-commands-v2-35","task":"","dataset":"Google Speech Commands V2 35","model":"M2D/0.7","rank_in_archive_order":1,"of":1,"metrics":{"ACC":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/on-voxceleb1","task":"","dataset":"VoxCeleb1","model":"M2D/0.7","rank_in_archive_order":1,"of":1,"metrics":{"Acc":"96.3"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-audio-set","task":"Audio Classification","dataset":"Audio Set","model":"M2D-AS/0.7","rank_in_archive_order":1,"of":3,"metrics":{"Mean AP":"48.5"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"M2D-AS/0.7","rank_in_archive_order":22,"of":51,"metrics":{"Test mAP":"0.485"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"M2D/0.7","rank_in_archive_order":27,"of":51,"metrics":{"Test mAP":"0.479"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"M2D-AS/0.7","rank_in_archive_order":9,"of":29,"metrics":{"Accuracy (5-fold)":"97.2","PRE-TRAINING DATASET":"AudioSet","Top-1 Accuracy":"97.2"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"M2D/0.7","rank_in_archive_order":14,"of":29,"metrics":{"Accuracy (5-fold)":"96.0","Top-1 Accuracy":"96.0"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-icbhi-respiratory","task":"Audio Classification","dataset":"ICBHI Respiratory Sound Database","model":"M2D-X/0.7 (η=0.3)","rank_in_archive_order":5,"of":25,"metrics":{"ICBHI Score":"63.29"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-icbhi-respiratory","task":"Audio Classification","dataset":"ICBHI Respiratory Sound Database","model":"M2D/0.7 (e=0.3)","rank_in_archive_order":7,"of":25,"metrics":{"ICBHI Score":"62.73"},"uses_additional_data":false},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"MSM-MAE","rank_in_archive_order":1,"of":12,"metrics":{"Accuracy":"96.6","Top-1 (%)":"96.6"},"uses_additional_data":true},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"M2D/0.6","rank_in_archive_order":2,"of":12,"metrics":{"Accuracy":"96.5","Top-1 (%)":"96.5"},"uses_additional_data":true},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"M2D/0.7","rank_in_archive_order":3,"of":12,"metrics":{"Accuracy":"96.3","Top-1 (%)":"96.3"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.06095","atlas_url":"https://app.syntology.ai/?focus=2404.06095","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06095"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nttcslab/m2d","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nttcslab/eval-audio-repr","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"44e9469142ae7a17","entry":"available_tasks","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"summarize.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/summarize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"44e9469142ae7a17"}},{"code_sha256_prefix":"ea9e4f4392740932","entry":"get_cache_info","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"lineareval.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/lineareval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ea9e4f4392740932"}},{"code_sha256_prefix":"dc2b7e0afed11044","entry":"is_zeroshot_ready","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"zeroshot.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dc2b7e0afed11044"}},{"code_sha256_prefix":"64321fd5ab7f96bc","entry":"load_pretrained_weights","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"evar/model_utils.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/evar/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"64321fd5ab7f96bc"}},{"code_sha256_prefix":"819b9385b29b2563","entry":"mean_max_pooling","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"evar/model_utils.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/evar/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"819b9385b29b2563"}},{"code_sha256_prefix":"14c8251e8baf656b","entry":"normalize_spectrogram","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"evar/ar_base.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/evar/ar_base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"14c8251e8baf656b"}},{"code_sha256_prefix":"94d3057985c9adeb","entry":"show_layers_trainable","repo":"nttcslab/eval-audio-repr","repo_kind":"official","path":"evar/model_utils.py","file_url":"https://github.com/nttcslab/eval-audio-repr/blob/HEAD/evar/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"94d3057985c9adeb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}