{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/delt-a-simple-diversity-driven-earlylate","title":"DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation","arxiv_id":"2411.19946","date":"2024-11-29","proceeding":"CVPR 2025 1","authors":["Zhiqiang Shen","Ammar Sherif","Zeyuan Yin","Shitong Shao"],"abstract":"Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new Diversity-driven EarlyLate Training (DELT) scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conduct extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our approach outperforms the previous state-of-the-art by 2$\\sim$5% on average across different datasets and IPCs (images per class), increasing diversity per class by more than 5% while reducing synthesis time by up to 39.3% for enhancing the training efficiency. Code is available at: https://github.com/VILA-Lab/DELT.","url_abs":"https://arxiv.org/abs/2411.19946v1","url_pdf":"https://arxiv.org/pdf/2411.19946v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"delt-a-simple-diversity-driven-earlylate","repo_url":"https://github.com/vila-lab/delt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dataset-distillation","task_name":"Dataset Distillation"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.19946","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.19946"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vila-lab/delt","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"eab4f74c38cc3c8f","entry":"accuracy","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/validation/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/validation/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eab4f74c38cc3c8f"}},{"code_sha256_prefix":"b615cefacfc04d10","entry":"batched_forward","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/model_utils/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/model_utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b615cefacfc04d10"}},{"code_sha256_prefix":"d276004a69566d12","entry":"clip","repo":"vila-lab/delt","repo_kind":"official","path":"recover/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d276004a69566d12"}},{"code_sha256_prefix":"70d1e8cd2526f3ae","entry":"denormalize","repo":"vila-lab/delt","repo_kind":"official","path":"recover/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70d1e8cd2526f3ae"}},{"code_sha256_prefix":"465f5964f29eaca8","entry":"get_dataloader","repo":"vila-lab/delt","repo_kind":"official","path":"data_selection.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/data_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"465f5964f29eaca8"}},{"code_sha256_prefix":"3724aa6b9357afef","entry":"get_img_indx","repo":"vila-lab/delt","repo_kind":"official","path":"recover/recover.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/recover.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3724aa6b9357afef"}},{"code_sha256_prefix":"ca0116cbf72e048c","entry":"get_mean_std","repo":"vila-lab/delt","repo_kind":"official","path":"recover/recover.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/recover.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca0116cbf72e048c"}},{"code_sha256_prefix":"ef8f26a5e81b47e2","entry":"get_model","repo":"vila-lab/delt","repo_kind":"official","path":"data_selection.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/data_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef8f26a5e81b47e2"}},{"code_sha256_prefix":"9a700a3b149f4952","entry":"get_parameters","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/validation/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/validation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a700a3b149f4952"}},{"code_sha256_prefix":"b7f0b1c1ecd5d257","entry":"get_round_imgs_indx","repo":"vila-lab/delt","repo_kind":"official","path":"recover/recover.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/recover.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7f0b1c1ecd5d257"}},{"code_sha256_prefix":"b76a08bec5e010fd","entry":"get_state_dict","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/model_utils/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/model_utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b76a08bec5e010fd"}},{"code_sha256_prefix":"c209acc6c3e5877a","entry":"keep_top_k","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/validation/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/validation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c209acc6c3e5877a"}},{"code_sha256_prefix":"952e2a07853f7c8e","entry":"load_model","repo":"vila-lab/delt","repo_kind":"official","path":"recover/models.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"952e2a07853f7c8e"}},{"code_sha256_prefix":"680f3862844d8a00","entry":"pad","repo":"vila-lab/delt","repo_kind":"official","path":"evaluation/model_utils/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/evaluation/model_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"680f3862844d8a00"}},{"code_sha256_prefix":"c80674a8056b7f9e","entry":"rank","repo":"vila-lab/delt","repo_kind":"official","path":"data_selection.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/data_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c80674a8056b7f9e"}},{"code_sha256_prefix":"e692498fea6f44ff","entry":"tiny_clip","repo":"vila-lab/delt","repo_kind":"official","path":"recover/utils.py","file_url":"https://github.com/vila-lab/delt/blob/HEAD/recover/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e692498fea6f44ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}