{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/synthio-augmenting-small-scale-audio","title":"Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data","arxiv_id":"2410.02056","date":"2024-10-02","proceeding":null,"authors":["Sreyan Ghosh","Sonal Kumar","Zhifeng Kong","Rafael Valle","Bryan Catanzaro","Dinesh Manocha"],"abstract":"We present Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data. Traditional data augmentation techniques, which apply artificial transformations (e.g., adding random noise or masking segments), struggle to create data that captures the true diversity present in real-world audios. To address this shortcoming, we propose to augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmentations is challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have sufficient compositional diversity. To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference optimization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models to (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used to prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captioned AudioSet.","url_abs":"https://arxiv.org/abs/2410.02056v2","url_pdf":"https://arxiv.org/pdf/2410.02056v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"synthio-augmenting-small-scale-audio","repo_url":"https://github.com/sreyan88/synthio","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.02056","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02056"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sreyan88/synthio","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"ran":6},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b0e0450aa0cadb7c","entry":"float32_to_int16_torch","repo":"sreyan88/synthio","repo_kind":"official","path":"gama_csv_inf.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/gama_csv_inf.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0e0450aa0cadb7c"}},{"code_sha256_prefix":"0d1f6516e19b52d6","entry":"get_flattened_caption_features_by_label","repo":"sreyan88/synthio","repo_kind":"official","path":"generate_captions_gpt.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/generate_captions_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d1f6516e19b52d6"}},{"code_sha256_prefix":"20a58a8c04ccda43","entry":"int16_to_float32_torch","repo":"sreyan88/synthio","repo_kind":"official","path":"gama_csv_inf.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/gama_csv_inf.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a58a8c04ccda43"}},{"code_sha256_prefix":"c15efdd12442aeb4","entry":"lookup_list","repo":"sreyan88/synthio","repo_kind":"official","path":"ast/src/dataloader.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/ast/src/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c15efdd12442aeb4"}},{"code_sha256_prefix":"a8fb3f62e2a901d7","entry":"make_index_dict","repo":"sreyan88/synthio","repo_kind":"official","path":"ast/src/dataloader.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/ast/src/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8fb3f62e2a901d7"}},{"code_sha256_prefix":"cbf115a314bcd23e","entry":"make_name_dict","repo":"sreyan88/synthio","repo_kind":"official","path":"ast/src/dataloader.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/ast/src/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbf115a314bcd23e"}},{"code_sha256_prefix":"90ab73aed24ee1ee","entry":"prompt_gpt","repo":"sreyan88/synthio","repo_kind":"official","path":"generate_captions_gpt.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/generate_captions_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90ab73aed24ee1ee"}},{"code_sha256_prefix":"6e560f4db57cde1f","entry":"select_half_items","repo":"sreyan88/synthio","repo_kind":"official","path":"generate_captions_gpt.py","file_url":"https://github.com/sreyan88/synthio/blob/HEAD/generate_captions_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e560f4db57cde1f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}