{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tuning-language-models-as-training-data","title":"Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning","arxiv_id":"2211.03044","date":"2022-11-06","proceeding":null,"authors":["Yu Meng","Martin Michalski","Jiaxin Huang","Yu Zhang","Tarek Abdelzaher","Jiawei Han"],"abstract":"Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few-shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points.","url_abs":"https://arxiv.org/abs/2211.03044v2","url_pdf":"https://arxiv.org/pdf/2211.03044v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tuning-language-models-as-training-data","repo_url":"https://github.com/yumeng5/fewgen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.03044","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.03044"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yumeng5/fewgen","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e448786d5ff1875a","entry":"default_dev_objective","repo":"yumeng5/fewgen","repo_kind":"official","path":"src/classification_trainer.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/src/classification_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e448786d5ff1875a"}},{"code_sha256_prefix":"126ae8eeb63d59bf","entry":"get_label","repo":"yumeng5/fewgen","repo_kind":"official","path":"utils/gen_k_shot.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/utils/gen_k_shot.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"126ae8eeb63d59bf"}},{"code_sha256_prefix":"e768e25ca95f70cd","entry":"input_example_to_string","repo":"yumeng5/fewgen","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e768e25ca95f70cd"}},{"code_sha256_prefix":"fbef70dc7c700501","entry":"input_example_to_tuple","repo":"yumeng5/fewgen","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fbef70dc7c700501"}},{"code_sha256_prefix":"8e27652a29c8adcc","entry":"load_datasets","repo":"yumeng5/fewgen","repo_kind":"official","path":"utils/gen_k_shot.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/utils/gen_k_shot.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e27652a29c8adcc"}},{"code_sha256_prefix":"efd1b7718b950470","entry":"split_header","repo":"yumeng5/fewgen","repo_kind":"official","path":"utils/gen_k_shot.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/utils/gen_k_shot.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efd1b7718b950470"}},{"code_sha256_prefix":"8b405662894bddba","entry":"combine","repo":"yumeng5/fewgen","repo_kind":"official","path":"utils/gen_utils.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/utils/gen_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b405662894bddba"}},{"code_sha256_prefix":"0d9c73d656213f02","entry":"read_files","repo":"yumeng5/fewgen","repo_kind":"official","path":"utils/gen_utils.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/utils/gen_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d9c73d656213f02"}},{"code_sha256_prefix":"7d99ecf76c902dc4","entry":"tokenize_multipart_input_classification","repo":"yumeng5/fewgen","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yumeng5/fewgen/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d99ecf76c902dc4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}