{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sslam-enhancing-self-supervised-models-with-1","title":"SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes","arxiv_id":"2506.12222","date":"2025-06-13","proceeding":"ICLR 2025 4","authors":["Tony Alex","Sara Ahmed","Armin Mustafa","Muhammad Awais","Philip JB Jackson"],"abstract":"Self-supervised pre-trained audio networks have seen widespread adoption in real-world systems, particularly in multi-modal large language models. These networks are often employed in a frozen state, under the assumption that the SSL pre-training has sufficiently equipped them to handle real-world audio. However, a critical question remains: how well do these models actually perform in real-world conditions, where audio is typically polyphonic and complex, involving multiple overlapping sound sources? Current audio SSL methods are often benchmarked on datasets predominantly featuring monophonic audio, such as environmental sounds, and speech. As a result, the ability of SSL models to generalize to polyphonic audio, a common characteristic in natural scenarios, remains underexplored. This limitation raises concerns about the practical robustness of SSL models in more realistic audio settings. To address this gap, we introduce Self-Supervised Learning from Audio Mixtures (SSLAM), a novel direction in audio SSL research, designed to improve, designed to improve the model's ability to learn from polyphonic data while maintaining strong performance on monophonic data. We thoroughly evaluate SSLAM on standard audio SSL benchmark datasets which are predominantly monophonic and conduct a comprehensive comparative analysis against SOTA methods using a range of high-quality, publicly available polyphonic datasets. SSLAM not only improves model performance on polyphonic audio, but also maintains or exceeds performance on standard audio SSL benchmarks. Notably, it achieves up to a 3.9\\% improvement on the AudioSet-2M (AS-2M), reaching a mean average precision (mAP) of 50.2. For polyphonic datasets, SSLAM sets new SOTA in both linear evaluation and fine-tuning regimes with performance improvements of up to 9.1\\% (mAP).","url_abs":"https://arxiv.org/abs/2506.12222v1","url_pdf":"https://arxiv.org/pdf/2506.12222v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sslam-enhancing-self-supervised-models-with-1","repo_url":"https://github.com/ta012/SSLAM","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"SSLAM (Audio-Only, Single)","rank_in_archive_order":9,"of":51,"metrics":{"Test mAP":"0.502"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-balanced-audio-set","task":"Audio Classification","dataset":"Balanced Audio Set","model":"SSLAM","rank_in_archive_order":2,"of":8,"metrics":{"Mean AP":"40.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.12222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.12222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ta012/SSLAM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"named_in_paper":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"77e8a3ac46f3afec","entry":"get_2d_sincos_pos_embed","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/mae.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/mae.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77e8a3ac46f3afec"}},{"code_sha256_prefix":"68fc090d0a502993","entry":"get_2d_sincos_pos_embed_flexible","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/mae.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/mae.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68fc090d0a502993"}},{"code_sha256_prefix":"5d5be3c7916a67cb","entry":"get_annealed_rate","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/base.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d5be3c7916a67cb"}},{"code_sha256_prefix":"3f903f847b8f7f28","entry":"get_layer_id_for_vit","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/EAT_audio_classification.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/EAT_audio_classification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f903f847b8f7f28"}},{"code_sha256_prefix":"ee9ac852b4eb7975","entry":"random_masking","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/base.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee9ac852b4eb7975"}},{"code_sha256_prefix":"3a95f65cb24c84e6","entry":"gather_unmasked","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/base.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a95f65cb24c84e6"}},{"code_sha256_prefix":"915281feeb90ee5a","entry":"modify_relative_position_bias","repo":"ta012/SSLAM","repo_kind":"named_in_paper","path":"SSLAM_Inference/models/mae.py","file_url":"https://github.com/ta012/SSLAM/blob/HEAD/SSLAM_Inference/models/mae.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"915281feeb90ee5a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}