{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distributional-dataset-distillation-with","title":"Distributional Dataset Distillation with Subtask Decomposition","arxiv_id":"2403.00999","date":"2024-03-01","proceeding":null,"authors":["Tian Qin","Zhiwei Deng","David Alvarez-Melis"],"abstract":"What does a neural network learn when training from a task-specific dataset? Synthesizing this knowledge is the central idea behind Dataset Distillation, which recent work has shown can be used to compress large datasets into a small set of input-label pairs ($\\textit{prototypes}$) that capture essential aspects of the original dataset. In this paper, we make the key observation that existing methods distilling into explicit prototypes are very often suboptimal, incurring in unexpected storage cost from distilled labels. In response, we propose $\\textit{Distributional Dataset Distillation}$ (D3), which encodes the data using minimal sufficient per-class statistics and paired with a decoder, we distill dataset into a compact distributional representation that is more memory-efficient compared to prototype-based methods. To scale up the process of learning these representations, we propose $\\textit{Federated distillation}$, which decomposes the dataset into subsets, distills them in parallel using sub-task experts and then re-aggregates them. We thoroughly evaluate our algorithm on a three-dimensional metric and show that our method achieves state-of-the-art results on TinyImageNet and ImageNet-1K. Specifically, we outperform the prior art by $6.9\\%$ on ImageNet-1K under the storage budget of 2 images per class.","url_abs":"https://arxiv.org/abs/2403.00999v1","url_pdf":"https://arxiv.org/pdf/2403.00999v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distributional-dataset-distillation-with","repo_url":"https://github.com/sunnytqin/d3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"dataset-distillation","task_name":"Dataset Distillation"},{"task_slug":"decoder","task_name":"Decoder"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.00999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.00999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sunnytqin/d3","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"369e271a765fa529","entry":"VGG11","repo":"sunnytqin/d3","repo_kind":"official","path":"networks.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"369e271a765fa529"}},{"code_sha256_prefix":"f0e67400e8e03159","entry":"get_default_convnet_setting","repo":"sunnytqin/d3","repo_kind":"official","path":"utils.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f0e67400e8e03159"}},{"code_sha256_prefix":"bb890357a4a87c70","entry":"linear_mmd2","repo":"sunnytqin/d3","repo_kind":"official","path":"mmd.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/mmd.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb890357a4a87c70"}},{"code_sha256_prefix":"9970f30509d9c6ff","entry":"poly_mmd2","repo":"sunnytqin/d3","repo_kind":"official","path":"mmd.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/mmd.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9970f30509d9c6ff"}},{"code_sha256_prefix":"862e41e6a130a650","entry":"VGG11BN","repo":"sunnytqin/d3","repo_kind":"official","path":"networks.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"862e41e6a130a650"}},{"code_sha256_prefix":"c7c91cb61491f597","entry":"VGG13","repo":"sunnytqin/d3","repo_kind":"official","path":"networks.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c7c91cb61491f597"}},{"code_sha256_prefix":"95ba2cb15ce11fa9","entry":"get_dataset","repo":"sunnytqin/d3","repo_kind":"official","path":"utils.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"95ba2cb15ce11fa9"}},{"code_sha256_prefix":"b966e6d574382f69","entry":"mix_rbf_mmd2","repo":"sunnytqin/d3","repo_kind":"official","path":"mmd.py","file_url":"https://github.com/sunnytqin/d3/blob/HEAD/mmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b966e6d574382f69"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}