{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/synthetic-data-real-errors-how-not-to-publish","title":"Synthetic data, real errors: how (not) to publish and use synthetic data","arxiv_id":"2305.09235","date":"2023-05-16","proceeding":null,"authors":["Boris van Breugel","Zhaozhi Qian","Mihaela van der Schaar"],"abstract":"Generating synthetic data through generative models is gaining interest in the ML community and beyond, promising a future where datasets can be tailored to individual needs. Unfortunately, synthetic data is usually not perfect, resulting in potential errors in downstream tasks. In this work we explore how the generative process affects the downstream ML task. We show that the naive synthetic data approach -- using synthetic data as if it is real -- leads to downstream models and analyses that do not generalize well to real data. As a first step towards better ML in the synthetic data regime, we introduce Deep Generative Ensemble (DGE) -- a framework inspired by Deep Ensembles that aims to implicitly approximate the posterior distribution over the generative process model parameters. DGE improves downstream model training, evaluation, and uncertainty quantification, vastly outperforming the naive approach on average. The largest improvements are achieved for minority classes and low-density regions of the original data, for which the generative uncertainty is largest.","url_abs":"https://arxiv.org/abs/2305.09235v2","url_pdf":"https://arxiv.org/pdf/2305.09235v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"synthetic-data-real-errors-how-not-to-publish","repo_url":"https://github.com/bvanbreugel/deep_generative_ensemble","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"deep-ensembles","method_name":"Deep Ensembles"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.09235","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.09235"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bvanbreugel/deep_generative_ensemble","reach":null}],"summary":{"ran":1,"ran_fixture":2,"ran_draft_wrong":9,"ran_honours":1,"ran_violates":1,"unverified":8},"by_repo_kind":{"official":{"samples":22,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":22,"samples":[{"code_sha256_prefix":"e647842d12d6c658","entry":"Constraints","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e647842d12d6c658"}},{"code_sha256_prefix":"2b0705f900803af1","entry":"compress_dataset","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b0705f900803af1"}},{"code_sha256_prefix":"17a967132aae2f11","entry":"compute_metrics","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17a967132aae2f11"}},{"code_sha256_prefix":"e0c0041bd58cb299","entry":"create_log_and_print_function","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0c0041bd58cb299"}},{"code_sha256_prefix":"e22cc4901a4c10d7","entry":"dataframe_hash","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e22cc4901a4c10d7"}},{"code_sha256_prefix":"4a483d9850b21f9c","entry":"evaluate_classifier","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a483d9850b21f9c"}},{"code_sha256_prefix":"78203e7d48410bcd","entry":"evaluate_regression","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78203e7d48410bcd"}},{"code_sha256_prefix":"39ebf38d74f7ab03","entry":"generate_score","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39ebf38d74f7ab03"}},{"code_sha256_prefix":"6a6298cdd3dd37a3","entry":"init_model","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a6298cdd3dd37a3"}},{"code_sha256_prefix":"2cbef8646196cf06","entry":"load","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cbef8646196cf06"}},{"code_sha256_prefix":"e9b058f188a5ab6e","entry":"meanstd","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9b058f188a5ab6e"}},{"code_sha256_prefix":"3abf4dc0433d8b23","entry":"print_score","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3abf4dc0433d8b23"}},{"code_sha256_prefix":"9aba7ba5166d81ab","entry":"save","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9aba7ba5166d81ab"}},{"code_sha256_prefix":"fd1e2201eb28a95c","entry":"tt_predict_performance","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd1e2201eb28a95c"}},{"code_sha256_prefix":"e96e9dad554ff546","entry":"DataLoader","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e96e9dad554ff546"}},{"code_sha256_prefix":"5954d9082e003af8","entry":"GenericDataLoader","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5954d9082e003af8"}},{"code_sha256_prefix":"f78bfc394becad48","entry":"aggregate_predictive","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f78bfc394becad48"}},{"code_sha256_prefix":"16b7e80881d5bb79","entry":"cat_dl","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16b7e80881d5bb79"}},{"code_sha256_prefix":"85acfb6142849d42","entry":"decompress_dataset","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85acfb6142849d42"}},{"code_sha256_prefix":"847cac96db219025","entry":"enable_reproducible_results","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"847cac96db219025"}},{"code_sha256_prefix":"b253c16de5b20700","entry":"error","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b253c16de5b20700"}},{"code_sha256_prefix":"3150dee7f480c2a8","entry":"info","repo":"bvanbreugel/deep_generative_ensemble","repo_kind":"official","path":"DGE_utils.py","file_url":"https://github.com/bvanbreugel/deep_generative_ensemble/blob/HEAD/DGE_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3150dee7f480c2a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}