{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ast-audio-spectrogram-transformer","title":"AST: Audio Spectrogram Transformer","arxiv_id":"2104.01778","date":"2021-04-05","proceeding":null,"authors":["Yuan Gong","Yu-An Chung","James Glass"],"abstract":"In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.","url_abs":"https://arxiv.org/abs/2104.01778v3","url_pdf":"https://arxiv.org/pdf/2104.01778v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ast-audio-spectrogram-transformer","repo_url":"https://github.com/YuanGongND/ast","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"ast-audio-spectrogram-transformer","repo_url":"https://github.com/cgaroufis/msspt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"ast-audio-spectrogram-transformer","repo_url":"https://github.com/nttcslab/composing-general-audio-repr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"ast-audio-spectrogram-transformer","repo_url":"https://github.com/pxaris/ccml","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"ast-audio-spectrogram-transformer","repo_url":"https://github.com/pwc-1/Paper-8/tree/main/audio_spectrogram_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"audio-tagging","task_name":"Audio Tagging"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":"speech-emotion-recognition","task_name":"Speech Emotion Recognition"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"AST (Ensemble)","rank_in_archive_order":20,"of":51,"metrics":{"Test mAP":"0.485"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"AST (Single)","rank_in_archive_order":37,"of":51,"metrics":{"Test mAP":"0.459"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"Audio Spectrogram Transformer","rank_in_archive_order":16,"of":29,"metrics":{"Accuracy (5-fold)":"95.7","PRE-TRAINING DATASET":"AudioSet, ImageNet","Top-1 Accuracy":"95.7"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-speech-commands-1","task":"Audio Classification","dataset":"Speech Commands","model":"AST-S","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"98.11±0.05"},"uses_additional_data":false},{"leaderboard":"/sota/audio-tagging-on-audioset","task":"Audio Tagging","dataset":"AudioSet","model":"Audio Spectrogram Transformer","rank_in_archive_order":5,"of":11,"metrics":{"mean average precision":"0.485"},"uses_additional_data":true},{"leaderboard":"/sota/keyword-spotting-on-google-speech-commands","task":"Keyword Spotting","dataset":"Google Speech Commands","model":"Audio Spectrogram Transformer","rank_in_archive_order":32,"of":42,"metrics":{"Google Speech Commands V2 35":"98.11"},"uses_additional_data":false},{"leaderboard":"/sota/speech-emotion-recognition-on-crema-d","task":"Speech Emotion Recognition","dataset":"CREMA-D","model":"ViT","rank_in_archive_order":7,"of":9,"metrics":{"Accuracy":"67.81"},"uses_additional_data":false},{"leaderboard":"/sota/time-series-on-speech-commands","task":"Time Series Analysis","dataset":"Speech Commands","model":"ViT","rank_in_archive_order":2,"of":6,"metrics":{"% Test Accuracy":"98.11"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.01778","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.01778"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cgaroufis/msspt","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-8/tree/main/audio_spectrogram_transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nttcslab/composing-general-audio-repr","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YuanGongND/ast","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pxaris/ccml","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3eb071f1be404988","entry":"get_warmup_lr","repo":"pxaris/ccml","repo_kind":"listed","path":"training/utils.py","file_url":"https://github.com/pxaris/ccml/blob/HEAD/training/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3eb071f1be404988"}},{"code_sha256_prefix":"7116dd6a41d2b73f","entry":"opt_schedule","repo":"pxaris/ccml","repo_kind":"listed","path":"training/utils.py","file_url":"https://github.com/pxaris/ccml/blob/HEAD/training/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7116dd6a41d2b73f"}},{"code_sha256_prefix":"538dda2cb39fe247","entry":"train_one_epoch","repo":"pxaris/ccml","repo_kind":"listed","path":"training/utils.py","file_url":"https://github.com/pxaris/ccml/blob/HEAD/training/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"538dda2cb39fe247"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}