Papers › Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets
Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets
Paul Primus, Khaled Koutini, Gerhard Widmer
This work presents a text-to-audio-retrieval system based on pre-trained text and spectrogram transformers. Our method projects recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. Through a systematic analysis, we examine how each component of the system influences retrieval performance. As a result, we identify two key components that play a crucial role in driving performance: the self-attention-based audio encoder for audio embedding and the utilization of additional human-generated and synthetic data sets during pre-training. We further experimented with augmenting ClothoV2 captions with available keywords to increase their variety; however, this only led to marginal improvements. Our system ranked first in the 2023's DCASE Challenge, and it outperforms the current state of the art on the ClothoV2 benchmark by 5.6 pp. mAP@10.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Text to Audio Retrieval | Clotho | PaSST–RoBERTa & GPT-augment | R@1 | 26.07 | #4 of 12 | Archive leaderboard | report |
| Text to Audio Retrieval | Clotho | PaSST–RoBERTa & GPT-augment | R@10 | 69.30 | #4 of 12 | Archive leaderboard | report |
| Text to Audio Retrieval | Clotho | PaSST–RoBERTa & GPT-augment | R@5 | 55.27 | #4 of 12 | Archive leaderboard | report |
| Text to Audio Retrieval | Clotho | PaSST–RoBERTa & GPT-augment | mAP@10 | 38.56 | #4 of 12 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections