Papers › Zero-Shot Audio Captioning via Audibility Guidance
Zero-Shot Audio Captioning via Audibility Guidance
Tal Shaharabany, Ariel Shaulov, Lior Wolf
The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio -- (i) fluency of the generated text, (ii) faithfulness of the generated text to the input audio, and the somewhat related (iii) audibility, which is the quality of being able to be perceived based only on audio. Our method is a zero-shot method, i.e., we do not learn to perform captioning. Instead, captioning occurs as an inference process that involves three networks that correspond to the three desired qualities: (i) A Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A model that provides a matching score between an audio file and a text, for which we use a multimodal matching network called ImageBind, and (iii) A text classifier, trained using a dataset we collected automatically by instructing GPT-4 with prompts designed to direct the generation of both audible and inaudible sentences. We present our results on the AudioCap dataset, demonstrating that audibility guidance significantly enhances performance compared to the baseline, which lacks this objective.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Zero-shot Audio Captioning | AudioCaps | Shaharabany et al. | BLEU-4 | 9.8 | #3 of 4 | Archive leaderboard | report |
| Zero-shot Audio Captioning | AudioCaps | Shaharabany et al. | CIDEr | 9.2 | #3 of 4 | Archive leaderboard | report |
| Zero-shot Audio Captioning | AudioCaps | Shaharabany et al. | METEOR | 8.6 | #3 of 4 | Archive leaderboard | report |
| Zero-shot Audio Captioning | AudioCaps | Shaharabany et al. | ROUGE-L | 8.2 | #3 of 4 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections