{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-contrastive-language-audio","title":"Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation","arxiv_id":"2211.06687","date":"2022-11-12","proceeding":null,"authors":["Yusong Wu","Ke Chen","Tianyu Zhang","Yuchen Hui","Marianna Nezhurina","Taylor Berg-Kirkpatrick","Shlomo Dubnov"],"abstract":"Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.","url_abs":"https://arxiv.org/abs/2211.06687v4","url_pdf":"https://arxiv.org/pdf/2211.06687v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"large-scale-contrastive-language-audio","repo_url":"https://github.com/laion-ai/clap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"large-scale-contrastive-language-audio","repo_url":"https://github.com/gudgud96/frechet-audio-distance","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"large-scale-contrastive-language-audio","repo_url":"https://github.com/gzhu06/cacophony","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"large-scale-contrastive-language-audio","repo_url":"https://github.com/sreyan88/reclap","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.06687","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.06687"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/laion-ai/clap","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gudgud96/frechet-audio-distance","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sreyan88/reclap","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gzhu06/cacophony","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_violates":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"aae796df720d8f90","entry":"find_params_value","repo":"sreyan88/reclap","repo_kind":"listed","path":"train/src/laion_clap/evaluate/eval_retrieval.py","file_url":"https://github.com/sreyan88/reclap/blob/HEAD/train/src/laion_clap/evaluate/eval_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aae796df720d8f90"}},{"code_sha256_prefix":"f372ac173bd8f134","entry":"float32_to_int16","repo":"laion-ai/clap","repo_kind":"official","path":"src/laion_clap/unit_test.py","file_url":"https://github.com/laion-ai/clap/blob/HEAD/src/laion_clap/unit_test.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f372ac173bd8f134"}},{"code_sha256_prefix":"47846af09dcbdba8","entry":"int16_to_float32","repo":"laion-ai/clap","repo_kind":"official","path":"src/laion_clap/unit_test.py","file_url":"https://github.com/laion-ai/clap/blob/HEAD/src/laion_clap/unit_test.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47846af09dcbdba8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}