{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-language-models-as-synthetic-data","title":"Evaluating Language Models as Synthetic Data Generators","arxiv_id":"2412.03679","date":"2024-12-04","proceeding":null,"authors":["Seungone Kim","Juyoung Suk","Xiang Yue","Vijay Viswanathan","Seongyun Lee","Yizhong Wang","Kiril Gashteovski","Carolin Lawrence","Sean Welleck","Graham Neubig"],"abstract":"Given the increasing use of synthetic data in language model (LM) post-training, an LM's ability to generate high-quality data has become nearly as crucial as its ability to solve problems directly. While prior works have focused on developing effective data generation methods, they lack systematic comparison of different LMs as data generators in a unified setting. To address this gap, we propose AgoraBench, a benchmark that provides standardized settings and metrics to evaluate LMs' data generation abilities. Through synthesizing 1.26 million training instances using 6 LMs and training 99 student models, we uncover key insights about LMs' data generation capabilities. First, we observe that LMs exhibit distinct strengths. For instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet performs better at enhancing existing ones. Furthermore, our analysis reveals that an LM's data generation ability doesn't necessarily correlate with its problem-solving ability. Instead, multiple intrinsic features of data quality-including response quality, perplexity, and instruction difficulty-collectively serve as better indicators. Finally, we demonstrate that strategic choices in output format and cost-conscious model selection significantly impact data generation effectiveness.","url_abs":"https://arxiv.org/abs/2412.03679v1","url_pdf":"https://arxiv.org/pdf/2412.03679v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-language-models-as-synthetic-data","repo_url":"https://github.com/neulab/data-agora","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluating-language-models-as-synthetic-data","repo_url":"https://github.com/EduardoTrevino/fail-talms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.03679","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03679"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neulab/data-agora","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EduardoTrevino/fail-talms","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b490b98ef282ef0d","entry":"load_prompt_loader","repo":"neulab/data-agora","repo_kind":"official","path":"libs/data-agora/data_agora/core/prompt_loaders.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/libs/data-agora/data_agora/core/prompt_loaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b490b98ef282ef0d"}},{"code_sha256_prefix":"5dbecf8bd5f07213","entry":"process_json_files","repo":"neulab/data-agora","repo_kind":"official","path":"agora_scripts/upload_to_hf.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/agora_scripts/upload_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dbecf8bd5f07213"}},{"code_sha256_prefix":"2df3fcd731fa7c9c","entry":"setup_ray_spilling","repo":"neulab/data-agora","repo_kind":"official","path":"libs/data-agora/data_agora/serve.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/libs/data-agora/data_agora/serve.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2df3fcd731fa7c9c"}},{"code_sha256_prefix":"c19f3ea0c5effc23","entry":"validate_forbidden_keywords","repo":"neulab/data-agora","repo_kind":"official","path":"libs/data-agora/data_agora/core/validators.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/libs/data-agora/data_agora/core/validators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c19f3ea0c5effc23"}},{"code_sha256_prefix":"fc24d31cb1ef7c87","entry":"validate_keywords","repo":"neulab/data-agora","repo_kind":"official","path":"libs/data-agora/data_agora/core/validators.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/libs/data-agora/data_agora/core/validators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc24d31cb1ef7c87"}},{"code_sha256_prefix":"cdf51efdf4591ca6","entry":"validate_length","repo":"neulab/data-agora","repo_kind":"official","path":"libs/data-agora/data_agora/core/validators.py","file_url":"https://github.com/neulab/data-agora/blob/HEAD/libs/data-agora/data_agora/core/validators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdf51efdf4591ca6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}