{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/iterative-ensemble-training-with-anti","title":"Iterative Ensemble Training with Anti-Gradient Control for Mitigating Memorization in Diffusion Models","arxiv_id":"2407.15328","date":"2024-07-22","proceeding":null,"authors":["Xiao Liu","Xiaoliu Guan","Yu Wu","Jiaxu Miao"],"abstract":"Diffusion models, known for their tremendous ability to generate novel and high-quality samples, have recently raised concerns due to their data memorization behavior, which poses privacy risks. Recent approaches for memory mitigation either only focused on the text modality problem in cross-modal generation tasks or utilized data augmentation strategies. In this paper, we propose a novel training framework for diffusion models from the perspective of visual modality, which is more generic and fundamental for mitigating memorization. To facilitate ``forgetting'' of stored information in diffusion model parameters, we propose an iterative ensemble training strategy by splitting the data into multiple shards for training multiple models and intermittently aggregating these model parameters. Moreover, practical analysis of losses illustrates that the training loss for easily memorable images tends to be obviously lower. Thus, we propose an anti-gradient control method to exclude the sample with a lower loss value from the current mini-batch to avoid memorizing. Extensive experiments and analysis on \\crnote{four} datasets are conducted to illustrate the effectiveness of our method, and results show that our method successfully reduces memory capacity while even improving the performance slightly. Moreover, to save the computing cost, we successfully apply our method to fine-tune the well-trained diffusion models by limited epochs, demonstrating the applicability of our method. Code is available in https://github.com/liuxiao-guan/IET_AGC.","url_abs":"https://arxiv.org/abs/2407.15328v1","url_pdf":"https://arxiv.org/pdf/2407.15328v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"iterative-ensemble-training-with-anti","repo_url":"https://github.com/liuxiao-guan/iet_agc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"WTFPL"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.15328","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15328"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuxiao-guan/iet_agc","reach":{"status":"ok","spdx":"WTFPL"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"cd3edab30cfead44","entry":"calculate_frechet_distance","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"score/fid.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/score/fid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"cd3edab30cfead44"}},{"code_sha256_prefix":"68b817601e8d8589","entry":"custom_sort_key","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"attack/all_in.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/attack/all_in.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"68b817601e8d8589"}},{"code_sha256_prefix":"6a78f18d5b2ce056","entry":"extract","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"diffusion.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"6a78f18d5b2ce056"}},{"code_sha256_prefix":"9b81ae7906ed29eb","entry":"get_inception_score","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"score/inception_score.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/score/inception_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"9b81ae7906ed29eb"}},{"code_sha256_prefix":"3cd1ed6d022ffb34","entry":"maintain_array","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"attack/all_in.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/attack/all_in.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"3cd1ed6d022ffb34"}},{"code_sha256_prefix":"b304af5c4fc5341a","entry":"torch_cov","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"score/fid.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/score/fid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"b304af5c4fc5341a"}},{"code_sha256_prefix":"526c471ba9b04752","entry":"get_inception_and_fid_score","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"score/both.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/score/both.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"526c471ba9b04752"}},{"code_sha256_prefix":"2dfa8886e774088c","entry":"sqrt_newton_schulz","repo":"liuxiao-guan/iet_agc","repo_kind":"official","path":"score/fid.py","file_url":"https://github.com/liuxiao-guan/iet_agc/blob/HEAD/score/fid.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"2dfa8886e774088c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}