{"url":"/task/audio-captioning","name":"Audio captioning","slug":"audio-captioning","description_markdown":"Audio Captioning is the task of describing audio using text. The general approach is to use an audio encoder to encode the audio (example: PANN, CAV-MAE), and to use a decoder (example: transformer) to generate the text.\r\nTo judge the quality of audio captions, though machine translation metrics (BLEU, METEOR, ROUGE)  and image captioning metrics (SPICE, CIDER) are used, they are not very well-suited. Attempts have been made to use pretrained language model based metrics such as Sentence-BERT.","categories":[{"name":"Audio","url":"/area/audio"}],"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","slug_source":"archive_url"},"counts":{"papers_tagged":119,"papers_with_code":62,"benchmarks":2,"benchmark_tables_in_archive":2,"benchmark_tables_shown":2,"benchmark_tables_withheld_as_spam":0,"benchmark_definition":"a leaderboard table with at least one row; benchmark_tables_shown also counts the zero-row tables; benchmark_tables_in_archive adds the tables withheld as spam","datasets":5,"subtasks":2,"parent_tasks":0},"benchmarks":[{"leaderboard":"/sota/audio-captioning-on-audiocaps","slug":"audio-captioning-on-audiocaps","dataset":"AudioCaps","dataset_url":"/dataset/audiocaps","rows_in_archive":18,"metrics":["SPIDEr","CIDEr","SPICE","BLEU-4","METEOR","ROUGE-L","FENSE","SPIDEr-FL","#params (M)","ROUGE","Sentence-BERT"],"first_row_in_archive_order":{"model":"MQ-Cap","paper_title":"Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning","paper_url":"/paper/audio-captioning-via-generative-pair-to-pair","paper_date":"2024-10-14","arxiv_id":"2410.10913","code_links":[],"syntology":null}},{"leaderboard":"/sota/audio-captioning-on-clotho","slug":"audio-captioning-on-clotho","dataset":"Clotho","dataset_url":"/dataset/clotho","rows_in_archive":11,"metrics":["SPIDEr","CIDEr","SPICE","BLEU-4","METEOR","ROUGE-L","FENSE","SPIDEr-FL","Sentence-BERT"],"first_row_in_archive_order":{"model":"SLAM-AAC","paper_title":"SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs","paper_url":"/paper/slam-aac-enhancing-audio-captioning-with","paper_date":"2024-10-12","arxiv_id":"2410.09503","code_links":[{"title":"X-LANCE/SLAM-LLM","url":"https://github.com/X-LANCE/SLAM-LLM"}],"syntology":null}}],"datasets":[{"url":"/dataset/audiocaps","name":"AudioCaps","full_name":"","num_papers_in_archive":279},{"url":"/dataset/clotho","name":"Clotho","full_name":"Clotho","num_papers_in_archive":202},{"url":"/dataset/wavcaps","name":"WavCaps","full_name":"","num_papers_in_archive":41},{"url":"/dataset/macs","name":"MACS","full_name":"Multi-Annotator Captioned Soundscapes","num_papers_in_archive":10},{"url":"/dataset/longvale","name":"LongVALE","full_name":"","num_papers_in_archive":2}],"subtasks":[{"url":"/task/retrieval-augmented-few-shot-in-context-audio","name":"Retrieval-augmented Few-shot In-context Audio Captioning"},{"url":"/task/zero-shot-audio-captioning","name":"Zero-shot Audio Captioning"}],"parent_tasks":[],"papers":{"order":"repositories listed in the archive (desc), then date (desc); the archive holds no stars","population":"papers tagged with this task that list at least one repository in the archive","shown":30,"of":62,"tagged_in_all":119,"items":[{"url":"/paper/clotho-an-audio-captioning-dataset","title":"Clotho: An Audio Captioning Dataset","date":"2019-10-21","arxiv_id":"1910.09387","repositories_listed":7,"syntology":{"n":21,"n_ran":6,"n_unverified":15,"n_pointer_only":0}},{"url":"/paper/wavcaps-a-chatgpt-assisted-weakly-labelled","title":"WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research","date":"2023-03-30","arxiv_id":"2303.17395","repositories_listed":3,"syntology":{"n":12,"n_ran":0,"n_unverified":12,"n_pointer_only":0}},{"url":"/paper/m2d2-exploring-general-purpose-audio-language","title":"M2D2: Exploring General-purpose Audio-Language Representations Beyond CLAP","date":"2025-03-28","arxiv_id":"2503.22104","repositories_listed":2,"syntology":null},{"url":"/paper/lavcap-llm-based-audio-visual-captioning","title":"LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport","date":"2025-01-16","arxiv_id":"2501.09291","repositories_listed":2,"syntology":null},{"url":"/paper/understanding-sounds-missing-the-questions","title":"Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models","date":"2024-06-12","arxiv_id":"2406.08402","repositories_listed":2,"syntology":null},{"url":"/paper/qwen-audio-advancing-universal-audio","title":"Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models","date":"2023-11-14","arxiv_id":"2311.07919","repositories_listed":2,"syntology":{"n":7,"n_ran":5,"n_unverified":2,"n_pointer_only":7}},{"url":"/paper/lauragpt-listen-attend-understand-and","title":"LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT","date":"2023-10-07","arxiv_id":"2310.04673","repositories_listed":2,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/vast-a-vision-audio-subtitle-text-omni-1","title":"VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset","date":"2023-05-29","arxiv_id":"2305.18500","repositories_listed":2,"syntology":{"n":42,"n_ran":15,"n_unverified":27,"n_pointer_only":0}},{"url":"/paper/cl4ac-a-contrastive-loss-for-audio-captioning","title":"CL4AC: A Contrastive Loss for Audio Captioning","date":"2021-07-21","arxiv_id":"2107.09990","repositories_listed":2,"syntology":{"n":1,"n_ran":1,"n_unverified":0,"n_pointer_only":1}},{"url":"/paper/video-salmonn-2-captioning-enhanced-audio","title":"video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models","date":"2025-06-18","arxiv_id":"2506.15220","repositories_listed":1,"syntology":{"n":8,"n_ran":3,"n_unverified":5,"n_pointer_only":0}},{"url":"/paper/fusionaudio-1-2m-towards-fine-grained-audio","title":"FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion","date":"2025-06-01","arxiv_id":"2506.01111","repositories_listed":1,"syntology":null},{"url":"/paper/solla-towards-a-speech-oriented-llm-that","title":"Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context","date":"2025-03-19","arxiv_id":"2503.15338","repositories_listed":1,"syntology":null},{"url":"/paper/mellow-a-small-audio-language-model-for","title":"Mellow: a small audio language model for reasoning","date":"2025-03-11","arxiv_id":"2503.08540","repositories_listed":1,"syntology":{"n":6,"n_ran":3,"n_unverified":3,"n_pointer_only":0}},{"url":"/paper/adiff-explaining-audio-difference-using","title":"ADIFF: Explaining audio difference using natural language","date":"2025-02-06","arxiv_id":"2502.04476","repositories_listed":1,"syntology":{"n":4,"n_ran":3,"n_unverified":1,"n_pointer_only":0}},{"url":"/paper/llms-can-see-and-hear-without-any-training","title":"LLMs can see and hear without any training","date":"2025-01-30","arxiv_id":"2501.18096","repositories_listed":1,"syntology":null},{"url":"/paper/etta-elucidating-the-design-space-of-text-to","title":"ETTA: Elucidating the Design Space of Text-to-Audio Models","date":"2024-12-26","arxiv_id":"2412.19351","repositories_listed":1,"syntology":null},{"url":"/paper/audiosetcaps-an-enriched-audio-caption","title":"AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Models","date":"2024-11-28","arxiv_id":"2411.18953","repositories_listed":1,"syntology":{"n":4,"n_ran":0,"n_unverified":4,"n_pointer_only":4}},{"url":"/paper/tell-what-you-hear-from-what-you-see-video-to","title":"Tell What You Hear From What You See -- Video to Audio Generation Through Text","date":"2024-11-08","arxiv_id":"2411.05679","repositories_listed":1,"syntology":{"n":6,"n_ran":0,"n_unverified":6,"n_pointer_only":6}},{"url":"/paper/drcap-decoding-clap-latents-with-retrieval","title":"DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning","date":"2024-10-12","arxiv_id":"2410.09472","repositories_listed":1,"syntology":null},{"url":"/paper/slam-aac-enhancing-audio-captioning-with","title":"SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs","date":"2024-10-12","arxiv_id":"2410.09503","repositories_listed":1,"syntology":null},{"url":"/paper/an-eye-for-an-ear-zero-shot-audio-description","title":"An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment","date":"2024-10-08","arxiv_id":"2410.05997","repositories_listed":1,"syntology":null},{"url":"/paper/opensep-leveraging-large-language-models-with","title":"OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation","date":"2024-09-28","arxiv_id":"2409.19270","repositories_listed":1,"syntology":null},{"url":"/paper/clair-a-leveraging-large-language-models-to","title":"CLAIR-A: Leveraging Large Language Models to Judge Audio Captions","date":"2024-09-19","arxiv_id":"2409.12962","repositories_listed":1,"syntology":null},{"url":"/paper/enclap-analyzing-the-enclap-framework-for","title":"EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance","date":"2024-09-02","arxiv_id":"2409.01201","repositories_listed":1,"syntology":null},{"url":"/paper/taming-data-and-transformers-for-audio-1","title":"Taming Data and Transformers for Audio Generation","date":"2024-06-27","arxiv_id":"2406.19388","repositories_listed":1,"syntology":null},{"url":"/paper/enhancing-automated-audio-captioning-via","title":"Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding","date":"2024-06-19","arxiv_id":"2406.13275","repositories_listed":1,"syntology":{"n":8,"n_ran":5,"n_unverified":3,"n_pointer_only":0}},{"url":"/paper/2406-15487","title":"Improving Text-To-Audio Models with Synthetic Captions","date":"2024-06-18","arxiv_id":"2406.15487","repositories_listed":1,"syntology":{"n":4,"n_ran":3,"n_unverified":1,"n_pointer_only":4}},{"url":"/paper/audio-flamingo-a-novel-audio-language-model","title":"Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities","date":"2024-02-02","arxiv_id":"2402.01831","repositories_listed":1,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":3}},{"url":"/paper/enclap-combining-neural-audio-codec-and-audio","title":"EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning","date":"2024-01-31","arxiv_id":"2401.17690","repositories_listed":1,"syntology":null},{"url":"/paper/learning-audio-concepts-from-counterfactual","title":"Learning Audio Concepts from Counterfactual Natural Language","date":"2024-01-10","arxiv_id":"2401.04935","repositories_listed":1,"syntology":null}],"syntology_records":14,"syntology_note":"a paper without a record is not a recorded non-run: it may lack an arXiv id or simply be absent from the graph layer"},"description_links":{"kept":0,"unwrapped_to_text":0,"bare_urls_linked":0,"relative_images_dropped":0,"rule":"internal links are kept only when the target slug exists in the catalog"},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per-sample execution status on synthesized fixtures ('ran N of M samples'); not a correctness claim and not a ranking signal.","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; our fixture could not drive it","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"}},"not_shown":{"libraries":"the archive has no per-task library table","trend_sparklines":"the Trend column of the benchmarks table was a rendered image; it is not in the archive","social_and_latest_sorts":"stars and social signals are not in the archive"}}