{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beats-audio-pre-training-with-acoustic","title":"BEATs: Audio Pre-Training with Acoustic Tokenizers","arxiv_id":"2212.09058","date":"2022-12-18","proceeding":null,"authors":["Sanyuan Chen","Yu Wu","Chengyi Wang","Shujie Liu","Daniel Tompkins","Zhuo Chen","Furu Wei"],"abstract":"The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.","url_abs":"https://arxiv.org/abs/2212.09058v1","url_pdf":"https://arxiv.org/pdf/2212.09058v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beats-audio-pre-training-with-acoustic","repo_url":"https://github.com/microsoft/unilm/tree/master/beats","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"beats-audio-pre-training-with-acoustic","repo_url":"https://github.com/Yui010206/CREMA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"beats-audio-pre-training-with-acoustic","repo_url":"https://github.com/phuriches/genrepasd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"beats-audio-pre-training-with-acoustic","repo_url":"https://github.com/qingyuliu0521/icsd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"BEATs (Audio-only, Ensemble)","rank_in_archive_order":7,"of":51,"metrics":{"Test mAP":"0.506"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"BEATs (Audio-only, Single)","rank_in_archive_order":17,"of":51,"metrics":{"Test mAP":"0.486"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-balanced-audio-set","task":"Audio Classification","dataset":"Balanced Audio Set","model":"BEATs","rank_in_archive_order":4,"of":8,"metrics":{"Mean AP":"38.9"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"BEATs","rank_in_archive_order":5,"of":29,"metrics":{"Accuracy (5-fold)":"98.1","PRE-TRAINING DATASET":"AudioSet","Top-1 Accuracy":"98.1"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.09058","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.09058"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yui010206/CREMA","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qingyuliu0521/icsd","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/unilm/tree/master/beats","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/phuriches/genrepasd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":15},"by_repo_kind":{"listed":{"samples":18,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f30ebf4e48b7e2d7","entry":"gelu","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/modules.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/modules.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f30ebf4e48b7e2d7"}},{"code_sha256_prefix":"eca6cdf05972a95e","entry":"l2norm","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/quantizer.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/quantizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eca6cdf05972a95e"}},{"code_sha256_prefix":"ada1d74afbd92171","entry":"sample_vectors","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/quantizer.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/quantizer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ada1d74afbd92171"}},{"code_sha256_prefix":"3d14ae419c0d7642","entry":"calc_dist_matrix","repo":"phuriches/genrepasd","repo_kind":"listed","path":"run_genrep_dcase2020.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/run_genrep_dcase2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d14ae419c0d7642"}},{"code_sha256_prefix":"4ca327b55579e45c","entry":"calc_dist_matrix","repo":"phuriches/genrepasd","repo_kind":"listed","path":"run_genrep_dcase2023.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/run_genrep_dcase2023.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ca327b55579e45c"}},{"code_sha256_prefix":"a7f6845c12992bc0","entry":"csv_text_to_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f6845c12992bc0"}},{"code_sha256_prefix":"e0640949f9aea446","entry":"eval_score","repo":"phuriches/genrepasd","repo_kind":"listed","path":"run_genrep_dcase2020.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/run_genrep_dcase2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0640949f9aea446"}},{"code_sha256_prefix":"4aa7333ffd7e18c6","entry":"gelu_accurate","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/modules.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4aa7333ffd7e18c6"}},{"code_sha256_prefix":"01233f4856456c5f","entry":"get_activation_fn","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/modules.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01233f4856456c5f"}},{"code_sha256_prefix":"3054478e3ed187fa","entry":"get_data_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/prepare_dcase2020.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/prepare_dcase2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3054478e3ed187fa"}},{"code_sha256_prefix":"f87d24572787fc14","entry":"get_data_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/prepare_dcase2023.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/prepare_dcase2023.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f87d24572787fc14"}},{"code_sha256_prefix":"8a194b39e1e24723","entry":"get_filename_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/prepare_dcase2020.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/prepare_dcase2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a194b39e1e24723"}},{"code_sha256_prefix":"8550a8f6ab58c11d","entry":"get_machines","repo":"phuriches/genrepasd","repo_kind":"listed","path":"dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8550a8f6ab58c11d"}},{"code_sha256_prefix":"81118c228b7322b3","entry":"get_meta_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/prepare_dcase2020.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/prepare_dcase2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81118c228b7322b3"}},{"code_sha256_prefix":"ed2f071daa13388b","entry":"get_meta_list","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/prepare_dcase2023.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/prepare_dcase2023.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2f071daa13388b"}},{"code_sha256_prefix":"7ff1101f9d23c903","entry":"get_section_ids","repo":"phuriches/genrepasd","repo_kind":"listed","path":"dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/dcase2023_task2_evaluator/dcase2023_task2_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ff1101f9d23c903"}},{"code_sha256_prefix":"b865ae3ee66b52d3","entry":"kmeans","repo":"phuriches/genrepasd","repo_kind":"listed","path":"beats/quantizer.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/beats/quantizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b865ae3ee66b52d3"}},{"code_sha256_prefix":"f975a9de7c08ff36","entry":"trainDataPct","repo":"phuriches/genrepasd","repo_kind":"listed","path":"src/datasets/utils.py","file_url":"https://github.com/phuriches/genrepasd/blob/HEAD/src/datasets/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f975a9de7c08ff36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}