{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unleash-data-generation-for-efficient-and","title":"NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation","arxiv_id":"2310.00258","date":"2023-09-30","proceeding":"CVPR 2024 1","authors":["Minh-Tuan Tran","Trung Le","Xuan-May Le","Mehrtash Harandi","Quan Hung Tran","Dinh Phung"],"abstract":"Data-Free Knowledge Distillation (DFKD) has made significant recent strides by transferring knowledge from a teacher neural network to a student neural network without accessing the original data. Nonetheless, existing approaches encounter a significant challenge when attempting to generate samples from random noise inputs, which inherently lack meaningful information. Consequently, these models struggle to effectively map this noise to the ground-truth sample distribution, resulting in prolonging training times and low-quality outputs. In this paper, we propose a novel Noisy Layer Generation method (NAYER) which relocates the random source from the input to a noisy layer and utilizes the meaningful constant label-text embedding (LTE) as the input. LTE is generated by using the language model once, and then it is stored in memory for all subsequent training processes. The significance of LTE lies in its ability to contain substantial meaningful inter-class information, enabling the generation of high-quality samples with only a few training steps. Simultaneously, the noisy layer plays a key role in addressing the issue of diversity in sample generation by preventing the model from overemphasizing the constrained label information. By reinitializing the noisy layer in each iteration, we aim to facilitate the generation of diverse samples while still retaining the method's efficiency, thanks to the ease of learning provided by LTE. Experiments carried out on multiple datasets demonstrate that our NAYER not only outperforms the state-of-the-art methods but also achieves speeds 5 to 15 times faster than previous approaches. The code is available at https://github.com/tmtuan1307/nayer.","url_abs":"https://arxiv.org/abs/2310.00258v2","url_pdf":"https://arxiv.org/pdf/2310.00258v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unleash-data-generation-for-efficient-and","repo_url":"https://github.com/tmtuan1307/nayer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"data-free-knowledge-distillation","task_name":"Data-free Knowledge Distillation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.00258","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00258"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tmtuan1307/nayer","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"5116e4067eb03c3e","entry":"cross_entropy","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/criterions.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5116e4067eb03c3e"}},{"code_sha256_prefix":"9e49766535e8f9e0","entry":"custom_cross_entropy","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/synthesis/nayer.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/synthesis/nayer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e49766535e8f9e0"}},{"code_sha256_prefix":"403277d0bfbb477b","entry":"find_classes","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/datasets/tiny_imagenet.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/datasets/tiny_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"403277d0bfbb477b"}},{"code_sha256_prefix":"f8610aafc779832f","entry":"jsdiv","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/criterions.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8610aafc779832f"}},{"code_sha256_prefix":"5418e150a16eff01","entry":"kldiv","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/criterions.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/criterions.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5418e150a16eff01"}},{"code_sha256_prefix":"5ded0879a886f3c4","entry":"make_dataset","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/datasets/tiny_imagenet.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/datasets/tiny_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ded0879a886f3c4"}},{"code_sha256_prefix":"d558beac7f489465","entry":"register_hooks","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/hooks.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/hooks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d558beac7f489465"}},{"code_sha256_prefix":"1b7f09e4bbd308e1","entry":"advarsarial_classification_evaluator","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/evaluators.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/evaluators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b7f09e4bbd308e1"}},{"code_sha256_prefix":"081369ab7a329c47","entry":"classification_evaluator","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/evaluators.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/evaluators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"081369ab7a329c47"}},{"code_sha256_prefix":"31a485e439cd35d5","entry":"segmentation_evaluator","repo":"tmtuan1307/nayer","repo_kind":"official","path":"datafree/evaluators.py","file_url":"https://github.com/tmtuan1307/nayer/blob/HEAD/datafree/evaluators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31a485e439cd35d5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}