Papers › SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine...

SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs

12 Oct 2024arXiv:2410.09503archive 2025-07-28

Wenxi Chen, Ziyang Ma, Xiquan Li, Xuenan Xu, Yuzhe Liang, Zhisheng Zheng, Kai Yu, Xie Chen

Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible. In this paper, we propose SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs. Our approach uses the self-supervised EAT model to extract fine-grained audio representations, which are then aligned with textual embeddings via lightweight linear layers. The caption generation LLM is efficiently fine-tuned using the LoRA adapter. Drawing inspiration from the back-translation method in machine translation, we implement paraphrasing augmentation to expand the Clotho dataset during pre-training. This strategy helps alleviate the limitation of scarce audio-text pairs and generates more diverse captions from a small set of audio clips. During inference, we introduce the plug-and-play CLAP-Refine strategy to fully exploit multiple decoding outputs, akin to the n-best rescoring strategy in speech recognition. Using the CLAP model for audio-text similarity calculation, we could select the textual descriptions generated by multiple searching beams that best match the input audio. Experimental results show that SLAM-AAC achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

X-LANCE/SLAM-LLM officialpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best... (arXiv:2601.09385; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Audio captioningCaption GenerationMachine TranslationSpeech RecognitionTranslationspeech-recognitiontext similarity

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Audio captioning AudioCaps SLAM-AAC CIDEr 0.841 #2 of 18 Archive leaderboard report
Audio captioning AudioCaps SLAM-AAC FENSE 0.668 #2 of 18 Archive leaderboard report
Audio captioning AudioCaps SLAM-AAC METEOR 0.268 #2 of 18 Archive leaderboard report
Audio captioning AudioCaps SLAM-AAC SPICE 0.194 #2 of 18 Archive leaderboard report
Audio captioning AudioCaps SLAM-AAC SPIDEr 0.518 #2 of 18 Archive leaderboard report
Audio captioning AudioCaps SLAM-AAC SPIDEr-FL 0.515 #2 of 18 Archive leaderboard report
Audio captioning Clotho SLAM-AAC CIDEr 0.515 #1 of 11 Archive leaderboard report
Audio captioning Clotho SLAM-AAC FENSE 0.540 #1 of 11 Archive leaderboard report
Audio captioning Clotho SLAM-AAC METEOR 0.197 #1 of 11 Archive leaderboard report
Audio captioning Clotho SLAM-AAC SPICE 0.148 #1 of 11 Archive leaderboard report
Audio captioning Clotho SLAM-AAC SPIDEr 0.332 #1 of 11 Archive leaderboard report
Audio captioning Clotho SLAM-AAC SPIDEr-FL 0.330 #1 of 11 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

SET

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections