{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/advancing-natural-language-based-audio","title":"Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets","arxiv_id":"2308.04258","date":"2023-08-08","proceeding":null,"authors":["Paul Primus","Khaled Koutini","Gerhard Widmer"],"abstract":"This work presents a text-to-audio-retrieval system based on pre-trained text and spectrogram transformers. Our method projects recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. Through a systematic analysis, we examine how each component of the system influences retrieval performance. As a result, we identify two key components that play a crucial role in driving performance: the self-attention-based audio encoder for audio embedding and the utilization of additional human-generated and synthetic data sets during pre-training. We further experimented with augmenting ClothoV2 captions with available keywords to increase their variety; however, this only led to marginal improvements. Our system ranked first in the 2023's DCASE Challenge, and it outperforms the current state of the art on the ClothoV2 benchmark by 5.6 pp. mAP@10.","url_abs":"https://arxiv.org/abs/2308.04258v1","url_pdf":"https://arxiv.org/pdf/2308.04258v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"advancing-natural-language-based-audio","repo_url":"https://github.com/optimusprimus/dcase2023_task6b","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-audio-retrieval-on-clotho","task":"Text to Audio Retrieval","dataset":"Clotho","model":"PaSST–RoBERTa & GPT-augment","rank_in_archive_order":4,"of":12,"metrics":{"R@1":"26.07","R@10":"69.30","R@5":"55.27","mAP@10":"38.56"},"uses_additional_data":true}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}