{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-pre-trained-models-assist-in-dataset","title":"Can pre-trained models assist in dataset distillation?","arxiv_id":"2310.03295","date":"2023-10-05","proceeding":null,"authors":["Yao Lu","Xuguang Chen","Yuchen Zhang","Jianyang Gu","Tianle Zhang","Yifan Zhang","Xiaoniu Yang","Qi Xuan","Kai Wang","Yang You"],"abstract":"Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.","url_abs":"https://arxiv.org/abs/2310.03295v1","url_pdf":"https://arxiv.org/pdf/2310.03295v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-pre-trained-models-assist-in-dataset","repo_url":"https://github.com/yaolu-zjut/ddinterpreter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dataset-distillation","task_name":"Dataset Distillation"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.03295","atlas_url":"https://app.syntology.ai/?focus=2310.03295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.03295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaolu-zjut/ddinterpreter","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"6e9c65aaa7c1b5da","entry":"distance_wb","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"methods/DC_DSA_DM/utils.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/methods/DC_DSA_DM/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e9c65aaa7c1b5da"}},{"code_sha256_prefix":"a2c08a19541743bb","entry":"get_dataset_info","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/Dataloader.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/Dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2c08a19541743bb"}},{"code_sha256_prefix":"8e1b179ef1f0fe03","entry":"VGG11","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/model.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e1b179ef1f0fe03"}},{"code_sha256_prefix":"cfab0fafbf988aa4","entry":"VGG11BN","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/model.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfab0fafbf988aa4"}},{"code_sha256_prefix":"07962bf28fe347a0","entry":"VGG13","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/model.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07962bf28fe347a0"}},{"code_sha256_prefix":"4ede4e38dc61074f","entry":"get_dataset","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"methods/DC_DSA_DM/utils.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/methods/DC_DSA_DM/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ede4e38dc61074f"}},{"code_sha256_prefix":"d6e228049010aa8c","entry":"get_dataset","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/Dataloader.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/Dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6e228049010aa8c"}},{"code_sha256_prefix":"ef0e8481768a2222","entry":"get_network","repo":"yaolu-zjut/ddinterpreter","repo_kind":"official","path":"utils_clom/model_pool.py","file_url":"https://github.com/yaolu-zjut/ddinterpreter/blob/HEAD/utils_clom/model_pool.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef0e8481768a2222"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}