{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-audio-captioning-models-with-fine","title":"Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation","arxiv_id":"2309.17352","date":"2023-09-29","proceeding":null,"authors":["Shih-Lun Wu","Xuankai Chang","Gordon Wichern","Jee-weon Jung","François Germain","Jonathan Le Roux","Shinji Watanabe"],"abstract":"Automated audio captioning (AAC) aims to generate informative descriptions for various sounds from nature and/or human activities. In recent years, AAC has quickly attracted research interest, with state-of-the-art systems now relying on a sequence-to-sequence (seq2seq) backbone powered by strong models such as Transformers. Following the macro-trend of applied machine learning research, in this work, we strive to improve the performance of seq2seq AAC models by extensively leveraging pretrained models and large language models (LLMs). Specifically, we utilize BEATs to extract fine-grained audio features. Then, we employ Instructor LLM to fetch text embeddings of captions, and infuse their language-modality knowledge into BEATs audio features via an auxiliary InfoNCE loss function. Moreover, we propose a novel data augmentation method that uses ChatGPT to produce caption mix-ups (i.e., grammatical and compact combinations of two captions) which, together with the corresponding audio mixtures, increase not only the amount but also the complexity and diversity of training data. During inference, we propose to employ nucleus sampling and a hybrid reranking algorithm, which has not been explored in AAC research. Combining our efforts, our model achieves a new state-of-the-art 32.6 SPIDEr-FL score on the Clotho evaluation split, and wins the 2023 DCASE AAC challenge.","url_abs":"https://arxiv.org/abs/2309.17352v2","url_pdf":"https://arxiv.org/pdf/2309.17352v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"improving-audio-captioning-models-with-fine","repo_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.17352","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.17352"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b76e5f7f5d14bd86","entry":"collect_audio_text_cosine_sims","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"reranking/encoder_rerank_sampling_outputs.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/reranking/encoder_rerank_sampling_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b76e5f7f5d14bd86"}},{"code_sha256_prefix":"f30ebf4e48b7e2d7","entry":"gelu","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"model/modules.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/model/modules.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f30ebf4e48b7e2d7"}},{"code_sha256_prefix":"cc98488959676d1e","entry":"generate_caption_for_audio","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"inference_sampling.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/inference_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc98488959676d1e"}},{"code_sha256_prefix":"76b744aeab651517","entry":"mask_along_axis","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"model/specaug.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/model/specaug.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76b744aeab651517"}},{"code_sha256_prefix":"b1a20d3e3e5ca99a","entry":"read_inference_csv","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a20d3e3e5ca99a"}},{"code_sha256_prefix":"9253071bb73fb8b2","entry":"reformat_to_coco","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"caption_evaluation_tools/eval_metrics.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/caption_evaluation_tools/eval_metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9253071bb73fb8b2"}},{"code_sha256_prefix":"76b6ceba370cc6b3","entry":"rerank_captions_with_cosine_sim","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"reranking/encoder_rerank_sampling_outputs.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/reranking/encoder_rerank_sampling_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76b6ceba370cc6b3"}},{"code_sha256_prefix":"bc55a35bc0483eb0","entry":"rerank_captions_with_nll","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"reranking/decoder_rerank_sampling_outputs.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/reranking/decoder_rerank_sampling_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc55a35bc0483eb0"}},{"code_sha256_prefix":"460bae18834bc026","entry":"time_warp","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"model/specaug.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/model/specaug.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"460bae18834bc026"}},{"code_sha256_prefix":"4aa7333ffd7e18c6","entry":"gelu_accurate","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"model/modules.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/model/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4aa7333ffd7e18c6"}},{"code_sha256_prefix":"01233f4856456c5f","entry":"get_activation_fn","repo":"slseanwu/beats-conformer-bart-audio-captioner","repo_kind":"official","path":"model/modules.py","file_url":"https://github.com/slseanwu/beats-conformer-bart-audio-captioner/blob/HEAD/model/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01233f4856456c5f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}