{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ssast-self-supervised-audio-spectrogram","title":"SSAST: Self-Supervised Audio Spectrogram Transformer","arxiv_id":"2110.09784","date":"2021-10-19","proceeding":null,"authors":["Yuan Gong","Cheng-I Jeff Lai","Yu-An Chung","James Glass"],"abstract":"Recently, neural networks based purely on self-attention, such as the Vision Transformer (ViT), have been shown to outperform deep learning models constructed with convolutional neural networks (CNNs) on various vision tasks, thus extending the success of Transformers, which were originally developed for language processing, to the vision domain. A recent study showed that a similar methodology can also be applied to the audio domain. Specifically, the Audio Spectrogram Transformer (AST) achieves state-of-the-art results on various audio classification benchmarks. However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST. This paper focuses on audio and speech classification, and aims to reduce the need for large amounts of labeled data for AST by leveraging self-supervised learning using unlabeled data. Specifically, we propose to pretrain the AST model with joint discriminative and generative masked spectrogram patch modeling (MSPM) using unlabeled audio from AudioSet and Librispeech. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, emotion recognition, and speaker identification. The proposed self-supervised framework significantly boosts AST performance on all tasks, with an average improvement of 60.9%, leading to similar or even better results than a supervised pretrained AST. To the best of our knowledge, it is the first patch-based self-supervised learning framework in the audio and speech domain, and also the first self-supervised learning framework for AST.","url_abs":"https://arxiv.org/abs/2110.09784v2","url_pdf":"https://arxiv.org/pdf/2110.09784v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ssast-self-supervised-audio-spectrogram","repo_url":"https://github.com/YuanGongND/ssast","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ssast-self-supervised-audio-spectrogram","repo_url":"https://github.com/YuanGongND/ast","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"ssast-self-supervised-audio-spectrogram","repo_url":"https://github.com/siavashshams/ssamba","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speaker-identification","task_name":"Speaker Identification"},{"task_slug":"spoken-command-recognition","task_name":"Spoken Command Recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-balanced-audio-set","task":"Audio Classification","dataset":"Balanced Audio Set","model":"SSAST-PATCH","rank_in_archive_order":6,"of":8,"metrics":{"Mean AP":"31.0"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-balanced-audio-set","task":"Audio Classification","dataset":"Balanced Audio Set","model":"SSAST-FRAME","rank_in_archive_order":7,"of":8,"metrics":{"Mean AP":"29.2"},"uses_additional_data":false},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"SSAST-FRAME","rank_in_archive_order":9,"of":12,"metrics":{"Accuracy":"80.8","Top-1 (%)":"80.8"},"uses_additional_data":true},{"leaderboard":"/sota/speaker-identification-on-voxceleb1","task":"Speaker Identification","dataset":"VoxCeleb1","model":"SSAST-PATCH","rank_in_archive_order":11,"of":12,"metrics":{"Accuracy":"64.2","Top-1 (%)":"64.2"},"uses_additional_data":true},{"leaderboard":"/sota/spoken-command-recognition-on-speech-command","task":"Spoken Command Recognition","dataset":"Speech Command v2","model":"SSAST-FRAME","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"98.1"},"uses_additional_data":false},{"leaderboard":"/sota/spoken-command-recognition-on-speech-command","task":"Spoken Command Recognition","dataset":"Speech Command v2","model":"SSAST-PATCH","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"98.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.09784","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.09784"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YuanGongND/ast","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YuanGongND/ssast","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/siavashshams/ssamba","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":10,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":15,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"571ae4f8e76e6bc8","entry":"calculate_stats","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/utilities/stats.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/utilities/stats.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"571ae4f8e76e6bc8"}},{"code_sha256_prefix":"5ee11f4af0d2fc1e","entry":"computeMatchmap","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/utilities/util.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/utilities/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5ee11f4af0d2fc1e"}},{"code_sha256_prefix":"e469cef4384ff73a","entry":"d_prime","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/utilities/stats.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/utilities/stats.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e469cef4384ff73a"}},{"code_sha256_prefix":"e217fb47bef3abe4","entry":"get_immediate_subdirectories","repo":"YuanGongND/ast","repo_kind":"listed","path":"egs/esc50/prep_esc50.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/egs/esc50/prep_esc50.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e217fb47bef3abe4"}},{"code_sha256_prefix":"ced6bc2bfdfe0059","entry":"get_sinusoid_encoding","repo":"YuanGongND/ssast","repo_kind":"official","path":"src/models/ast_models.py","file_url":"https://github.com/YuanGongND/ssast/blob/HEAD/src/models/ast_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ced6bc2bfdfe0059"}},{"code_sha256_prefix":"c15efdd12442aeb4","entry":"lookup_list","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/dataloader.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c15efdd12442aeb4"}},{"code_sha256_prefix":"a8fb3f62e2a901d7","entry":"make_index_dict","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/dataloader.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a8fb3f62e2a901d7"}},{"code_sha256_prefix":"ad2738e74fab6072","entry":"make_index_dict","repo":"YuanGongND/ast","repo_kind":"listed","path":"egs/audioset/gen_weight_file.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/egs/audioset/gen_weight_file.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ad2738e74fab6072"}},{"code_sha256_prefix":"cbf115a314bcd23e","entry":"make_name_dict","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/dataloader.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cbf115a314bcd23e"}},{"code_sha256_prefix":"546678fcbd2854db","entry":"matchmapSim","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/utilities/util.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/utilities/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"546678fcbd2854db"}},{"code_sha256_prefix":"ecb5ae1f3a358b96","entry":"validate_ensemble","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/traintest.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/traintest.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ecb5ae1f3a358b96"}},{"code_sha256_prefix":"0bed13389f791924","entry":"calc_recalls","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/utilities/util.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/utilities/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0bed13389f791924"}},{"code_sha256_prefix":"5ea88601f05850d6","entry":"get_immediate_files","repo":"YuanGongND/ast","repo_kind":"listed","path":"egs/esc50/prep_esc50.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/egs/esc50/prep_esc50.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5ea88601f05850d6"}},{"code_sha256_prefix":"1c75a85694e30f0b","entry":"load_label","repo":"YuanGongND/ast","repo_kind":"listed","path":"egs/audioset/inference.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/egs/audioset/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1c75a85694e30f0b"}},{"code_sha256_prefix":"340de5fa0cf72870","entry":"validate","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/traintest.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/traintest.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"340de5fa0cf72870"}},{"code_sha256_prefix":"731e6108d9bb81c8","entry":"validate_wa","repo":"YuanGongND/ast","repo_kind":"listed","path":"src/traintest.py","file_url":"https://github.com/YuanGongND/ast/blob/HEAD/src/traintest.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"731e6108d9bb81c8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}