{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/audioclip-extending-clip-to-image-text-and","title":"AudioCLIP: Extending CLIP to Image, Text and Audio","arxiv_id":"2106.13043","date":"2021-06-24","proceeding":null,"authors":["Andrey Guzhov","Federico Raue","Jörn Hees","Andreas Dengel"],"abstract":"In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets 68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.","url_abs":"https://arxiv.org/abs/2106.13043v1","url_pdf":"https://arxiv.org/pdf/2106.13043v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"audioclip-extending-clip-to-image-text-and","repo_url":"https://github.com/AndreyGuzhov/AudioCLIP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"audioclip-extending-clip-to-image-text-and","repo_url":"https://github.com/asteroid-team/torch-audiomentations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"audioclip-extending-clip-to-image-text-and","repo_url":"https://github.com/iver56/audiomentations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"audioclip-extending-clip-to-image-text-and","repo_url":"https://github.com/julirao/whisper_audio_classification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"environmental-sound-classification","task_name":"Environmental Sound Classification"},{"task_slug":"sound-classification","task_name":"Sound Classification"},{"task_slug":"zero-shot-environment-sound-classification","task_name":"Zero-Shot Environment Sound Classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/environmental-sound-classification-on-esc-50","task":"Environmental Sound Classification","dataset":"ESC-50","model":"AudioCLIP","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"97.15"},"uses_additional_data":false},{"leaderboard":"/sota/environmental-sound-classification-on","task":"Environmental Sound Classification","dataset":"UrbanSound8K","model":"AudioCLIP","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"90.07"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.13043","atlas_url":"https://app.syntology.ai/?focus=2106.13043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.13043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AndreyGuzhov/AudioCLIP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/asteroid-team/torch-audiomentations","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/julirao/whisper_audio_classification","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iver56/audiomentations","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"listed":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c18d1f2fc8cc8c3","entry":"decay_to_beta","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/augmentations/add_color_noise.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/augmentations/add_color_noise.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c18d1f2fc8cc8c3"}},{"code_sha256_prefix":"7e90b20c82fb03c4","entry":"find_audio_files","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/core/utils.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e90b20c82fb03c4"}},{"code_sha256_prefix":"7adfe52e0a0a847d","entry":"find_audio_files_in_paths","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/core/utils.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7adfe52e0a0a847d"}},{"code_sha256_prefix":"7db07f5e74e6c524","entry":"format_args","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/core/utils.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7db07f5e74e6c524"}},{"code_sha256_prefix":"b07b270a65ccaba2","entry":"get_shortest_class_fullname","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/core/serialization.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/core/serialization.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b07b270a65ccaba2"}},{"code_sha256_prefix":"7fee2825b85bfbe7","entry":"shorten_class_name","repo":"iver56/audiomentations","repo_kind":"listed","path":"audiomentations/core/serialization.py","file_url":"https://github.com/iver56/audiomentations/blob/HEAD/audiomentations/core/serialization.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fee2825b85bfbe7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}