Browse State-of-the-Art › Audio captioning
Audio captioning
62 papers with code · 2 benchmarks · 5 datasets archive 2025-07-28
Audio Captioning is the task of describing audio using text. The general approach is to use an audio encoder to encode the audio (example: PANN, CAV-MAE), and to use a decoder (example: transformer) to generate the text. To judge the quality of audio captions, though machine translation metrics (BLEU, METEOR, ROUGE) and image captioning metrics (SPICE, CIDER) are used, they are not very well-suited. Attempts have been made to use pretrained language model based metrics such as Sentence-BERT.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 1 newer paper was judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): AudioCaps (1 judged, 0 placed), Clotho (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| AudioCaps (18 rows) | MQ-Cap | Enhancing Retrieval-Augmented Audio Captioning with... | — | — | Compare |
| Clotho (11 rows) | SLAM-AAC | SLAM-AAC: Enhancing Audio Captioning with Paraphrasing... | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
5 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
2 subtasks in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 119 · with a code link: 62 · where Syntology ran a sample: 17 (16 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (17 of 119 tagged: 16 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 119, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
21 Oct 2019 7 repositories listed Syntology 19 ran (of which 0 constructed an object rather than computing a result; 16 with no instrument failure: 0 honoured, 0 violated, 16 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 21 harvested samples) · 5 pointer-only (licence)Audio captioning is the novel task of general audio content description using free text.
-
30 Mar 2023 3 repositories listed Syntology community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 12 harvested samples)To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
-
28 Mar 2025 2 repositories listedIn the second stage, it learns CLAP features using the audio features learned from the LLM-based embeddings.
-
16 Jan 2025 2 repositories listedLAVCap employs an optimal transport-based alignment loss to bridge the modality gap between audio and visual features, enabling more effective semantic extraction.
-
12 Jun 2024 2 repositories listedLarge audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks.
-
14 Nov 2023 2 repositories listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 7 pointer-only (licence)Recently, instruction-following audio-language models have received broad attention for audio interaction with humans.
-
7 Oct 2023 2 repositories listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text…
-
29 May 2023 2 repositories listed Syntology official (archive's flag): 12 ran · 35 ran (of which 4 constructed an object rather than computing a result; 25 with no instrument failure: 4 honoured, 1 violated, 20 with no contract checked; 10 where Syntology's instrument failed) · 7 unverified (of 42 harvested samples) · 8 pointer-only (licence)Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various…
-
21 Jul 2021 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip.
-
18 Jun 2025 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples) · 1 pointer-only (licence)Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding.
-
1 Jun 2025 1 repository listedHigh-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to…
-
19 Mar 2025 1 repository listedLarge Language Models (LLMs) have recently shown remarkable ability to process not only text but also multimodal inputs such as speech and audio.
-
11 Mar 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 1 pointer-only (licence)To train Mellow, we introduce ReasonAQA, a dataset designed to enhance audio-grounded reasoning in models.
-
6 Feb 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 1 pointer-only (licence)Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings.
-
30 Jan 2025 1 repository listedWe present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM.
-
26 Dec 2024 1 repository listedRecent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts.
-
28 Nov 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)In this paper, we propose an automated pipeline that integrates audio-language models for fine-grained content extraction, LLMs for synthetic caption generation, and a contrastive language-audio pretraining (CLAP)…
-
8 Nov 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 6 pointer-only (licence)Experiments show that when VATT is compared to existing video-to-audio generation methods in objective metrics, it achieves competitive performance when the audio caption is not provided.
-
12 Oct 2024 1 repository listedBy tailoring the text embedding support and the caption datastore to the target domain, DRCap acquires a robust ability to adapt to new domains in a training-free manner.
-
12 Oct 2024 1 repository listedRecent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible.
-
8 Oct 2024 1 repository listedIn this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner.
-
28 Sep 2024 1 repository listedAudio separation in real-world scenarios, where mixtures contain a variable number of sources, presents significant challenges due to limitations of existing models, such as over-separation, under-separation, and…
-
19 Sep 2024 1 repository listedThe Automated Audio Captioning (AAC) task asks models to generate natural language descriptions of an audio input.
-
2 Sep 2024 1 repository listedWe investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme.
-
27 Jun 2024 1 repository listedThe scalability of ambient sound generators is hindered by data scarcity, insufficient caption quality, and limited scalability in model architecture.
-
19 Jun 2024 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples)Automated audio captioning (AAC) is an audio-to-text task to describe audio contents in natural language.
-
18 Jun 2024 1 repository listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models.
-
2 Feb 2024 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs.
-
31 Jan 2024 1 repository listedWe also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model.
-
10 Jan 2024 1 repository listedConventional audio classification relied on predefined classes, lacking the ability to learn from free-form text.
-
21 Nov 2023 1 repository listed
-
14 Nov 2023 1 repository listed Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 17 harvested samples) · 17 pointer-only (licence)
-
20 Oct 2023 1 repository listed
-
21 Sep 2023 1 repository listed
-
18 Sep 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples) · 8 pointer-only (licence)
-
14 Sep 2023 1 repository listed
-
23 Aug 2023 1 repository listed
-
16 Jun 2023 1 repository listed
-
19 May 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 1 pointer-only (licence)
-
15 May 2023 1 repository listed
-
17 Apr 2023 1 repository listed
-
30 Mar 2023 1 repository listed
-
14 Nov 2022 1 repository listed
-
28 Oct 2022 1 repository listed
-
28 Sep 2022 1 repository listed
-
8 Jul 2022 1 repository listed
-
13 Jun 2022 1 repository listed
-
25 May 2022 1 repository listed
-
18 Apr 2022 1 repository listed
-
10 Jan 2022 1 repository listed
-
17 Dec 2021 1 repository listed
-
15 Nov 2021 1 repository listed
-
10 Oct 2021 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
5 Aug 2021 1 repository listed
-
21 Jul 2021 1 repository listed
-
16 Jul 2021 1 repository listed
-
6 Jul 2021 1 repository listed
-
24 Apr 2021 1 repository listed
-
21 Oct 2020 1 repository listed
-
9 Jul 2020 1 repository listed
-
6 Jul 2020 1 repository listed
-
31 May 2019 1 repository listed
-
AC/DC: LLM-based Audio Comprehension via Dialogue Continuation12 Jun 2025 0 repositories listed
-
CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer1 Jun 2025 0 repositories listed
-
Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning28 May 2025 0 repositories listed
-
TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining12 May 2025 0 repositories listed
-
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities6 Mar 2025 0 repositories listed
-
Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders21 Feb 2025 0 repositories listed
-
Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning8 Feb 2025 0 repositories listed
-
CosyAudio: Improving Audio Generation with Confidence Scores and Synthetic Captions28 Jan 2025 0 repositories listed
-
Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model13 Jan 2025 0 repositories listed
-
Classifier-Guided Captioning Across Modalities3 Jan 2025 0 repositories listed
-
EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation15 Oct 2024 0 repositories listed
-
14 Oct 2024 0 repositories listed
-
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization9 Oct 2024 0 repositories listed
-
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models17 Sep 2024 0 repositories listed
-
Towards Diverse and Efficient Audio Captioning via Diffusion Models14 Sep 2024 0 repositories listed
-
Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models10 Sep 2024 0 repositories listed
-
Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning2 Sep 2024 0 repositories listed
-
Audio Dialogues: Dialogues dataset for audio and music understanding11 Apr 2024 0 repositories listed
-
Improved Baselines for Data-efficient Perceptual Augmentation of LLMs20 Mar 2024 0 repositories listed
-
Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning20 Sep 2023 0 repositories listed
-
Audio Difference Learning for Audio Captioning15 Sep 2023 0 repositories listed
-
Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation6 Sep 2023 0 repositories listed
-
Generating Realistic Images from In-the-wild Sounds5 Sep 2023 0 repositories listed
-
Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?29 Aug 2023 0 repositories listed
-
20 Aug 2023 0 repositories listed
-
Improving Audio Caption Fluency with Automatic Error Correction16 Jun 2023 0 repositories listed
-
Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning30 May 2023 0 repositories listed
-
Efficient Audio Captioning Transformer with Patchout and Text Guidance6 Apr 2023 0 repositories listed
-
Towards Generating Diverse Audio Captions via Adversarial Training5 Dec 2022 0 repositories listed
-
Impact of visual assistance for automated audio captioning18 Nov 2022 0 repositories listed
-
15 Nov 2022 0 repositories listed
-
Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics12 Nov 2022 0 repositories listed
-
31 Oct 2022 0 repositories listed
-
Automated Audio Captioning via Fusion of Low- and High- Dimensional Features10 Oct 2022 0 repositories listed
-
Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity3 Oct 2022 0 repositories listed
-
Language-based Audio Retrieval Task in DCASE 2022 Challenge20 Sep 2022 0 repositories listed
-
An investigation on selecting audio pre-trained models for audio captioning12 Aug 2022 0 repositories listed
-
Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learning4 Jun 2022 0 repositories listed
Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections