{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-mitigating-architecture-overfitting","title":"Towards Mitigating Architecture Overfitting on Distilled Datasets","arxiv_id":"2309.04195","date":"2023-09-08","proceeding":null,"authors":["Xuyang Zhong","Chen Liu"],"abstract":"Dataset distillation methods have demonstrated remarkable performance for neural networks trained with very limited training data. However, a significant challenge arises in the form of \\textit{architecture overfitting}: the distilled training dataset synthesized by a specific network architecture (i.e., training network) generates poor performance when trained by other network architectures (i.e., test networks), especially when the test networks have a larger capacity than the training network. This paper introduces a series of approaches to mitigate this issue. Among them, DropPath renders the large model to be an implicit ensemble of its sub-networks, and knowledge distillation ensures each sub-network acts similarly to the small but well-performing teacher network. These methods, characterized by their smoothing effects, significantly mitigate architecture overfitting. We conduct extensive experiments to demonstrate the effectiveness and generality of our methods. Particularly, across various scenarios involving different tasks and different sizes of distilled data, our approaches significantly mitigate architecture overfitting. Furthermore, our approaches achieve comparable or even superior performance when the test network is larger than the training network.","url_abs":"https://arxiv.org/abs/2309.04195v2","url_pdf":"https://arxiv.org/pdf/2309.04195v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-mitigating-architecture-overfitting","repo_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dataset-distillation","task_name":"Dataset Distillation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.04195","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.04195"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"3d5c3585d162a761","entry":"ResNet18","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"model.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d5c3585d162a761"}},{"code_sha256_prefix":"85d94fdef1cf29c2","entry":"ResNet34","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"model.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85d94fdef1cf29c2"}},{"code_sha256_prefix":"cc62fb1a937874d2","entry":"drop_path","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"model.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc62fb1a937874d2"}},{"code_sha256_prefix":"5a495a3506d5fe4b","entry":"get_distilled_data_loader","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"utils.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a495a3506d5fe4b"}},{"code_sha256_prefix":"49fec08c0e9cb0d6","entry":"DiffAugment","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"utils.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49fec08c0e9cb0d6"}},{"code_sha256_prefix":"284f61197f58c52b","entry":"get_data_loader","repo":"cityu-mlo/mitigate_architecture_overfitting","repo_kind":"official","path":"utils.py","file_url":"https://github.com/cityu-mlo/mitigate_architecture_overfitting/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"284f61197f58c52b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}