{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/masked-modeling-duo-learning-representations","title":"Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input","arxiv_id":"2210.14648","date":"2022-10-26","proceeding":null,"authors":["Daisuke Niizumi","Daiki Takeuchi","Yasunori Ohishi","Noboru Harada","Kunio Kashino"],"abstract":"Masked Autoencoders is a simple yet powerful self-supervised learning method. However, it learns representations indirectly by reconstructing masked input patches. Several methods learn representations directly by predicting representations of masked patches; however, we think using all patches to encode training signal representations is suboptimal. We propose a new method, Masked Modeling Duo (M2D), that learns representations directly while obtaining training signals using only masked patches. In the M2D, the online network encodes visible patches and predicts masked patch representations, and the target network, a momentum encoder, encodes masked patches. To better predict target representations, the online network should model the input well, while the target network should also model it well to agree with online predictions. Then the learned representations should better model the input. We validated the M2D by learning general-purpose audio representations, and M2D set new state-of-the-art performance on tasks such as UrbanSound8K, VoxCeleb1, AudioSet20K, GTZAN, and SpeechCommandsV2. We additionally validate the effectiveness of M2D for images using ImageNet-1K in the appendix.","url_abs":"https://arxiv.org/abs/2210.14648v3","url_pdf":"https://arxiv.org/pdf/2210.14648v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"masked-modeling-duo-learning-representations","repo_url":"https://github.com/nttcslab/m2d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"audio-tagging","task_name":"Audio Tagging"},{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":null,"task_name":"Keyword Spotting on Google Speech Commands"},{"task_slug":"music-genre-classification","task_name":"Music Genre Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speaker-identification","task_name":"Speaker Identification"}],"methods":[{"method_slug":"m2d","method_name":"M2D"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/keyword-spotting-on-google-speech-commands","task":"Keyword Spotting","dataset":"Google Speech Commands","model":"M2D","rank_in_archive_order":30,"of":42,"metrics":{"Google Speech Commands V2 35":"98.5"},"uses_additional_data":false},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"M2D ratio=0.6","rank_in_archive_order":5,"of":12,"metrics":{"Accuracy":"94.8","Top-1 (%)":"94.8"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.14648","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.14648"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nttcslab/m2d","reach":null}],"summary":{"ran_draft_wrong":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"c50176ea992ce886","entry":"drop_non_model_weights","repo":"nttcslab/m2d","repo_kind":"official","path":"examples/portable_m2d.py","file_url":"https://github.com/nttcslab/m2d/blob/HEAD/examples/portable_m2d.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c50176ea992ce886"}},{"code_sha256_prefix":"5a1ee4014b30e027","entry":"expand_size","repo":"nttcslab/m2d","repo_kind":"official","path":"examples/portable_m2d.py","file_url":"https://github.com/nttcslab/m2d/blob/HEAD/examples/portable_m2d.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a1ee4014b30e027"}},{"code_sha256_prefix":"0851eabe8d2e8b7d","entry":"parse_sizes_by_name","repo":"nttcslab/m2d","repo_kind":"official","path":"examples/portable_m2d.py","file_url":"https://github.com/nttcslab/m2d/blob/HEAD/examples/portable_m2d.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0851eabe8d2e8b7d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}