{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-and-improving-the-training-dynamics","title":"Analyzing and Improving the Training Dynamics of Diffusion Models","arxiv_id":"2312.02696","date":"2023-12-05","proceeding":"CVPR 2024 1","authors":["Tero Karras","Miika Aittala","Jaakko Lehtinen","Janne Hellsten","Timo Aila","Samuli Laine"],"abstract":"Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets. In this paper, we identify and rectify several causes for uneven and ineffective training in the popular ADM diffusion model architecture, without altering its high-level structure. Observing uncontrolled magnitude changes and imbalances in both the network activations and weights over the course of training, we redesign the network layers to preserve activation, weight, and update magnitudes on expectation. We find that systematic application of this philosophy eliminates the observed drifts and imbalances, resulting in considerably better networks at equal computational complexity. Our modifications improve the previous record FID of 2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic sampling. As an independent contribution, we present a method for setting the exponential moving average (EMA) parameters post-hoc, i.e., after completing the training run. This allows precise tuning of EMA length without the cost of performing several training runs, and reveals its surprising interactions with network architecture, training time, and guidance.","url_abs":"https://arxiv.org/abs/2312.02696v2","url_pdf":"https://arxiv.org/pdf/2312.02696v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/nvlabs/edm2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/FutureXiang/edm2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/YichengDWu/tinyedm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/dopplerchase/cira-diff","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/lucidrains/ema-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/mmathew23/improved_edm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"analyzing-and-improving-the-training-dynamics","repo_url":"https://github.com/gohyojun15/ANT_diffusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"philosophy","task_name":"Philosophy"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-XXL","rank_in_archive_order":20,"of":52,"metrics":{"FID":"1.81","NFE":"126"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-XL","rank_in_archive_order":21,"of":52,"metrics":{"FID":"1.85","NFE":"126"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-L","rank_in_archive_order":22,"of":52,"metrics":{"FID":"1.88","NFE":"126"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-M","rank_in_archive_order":27,"of":52,"metrics":{"FID":"2.01","NFE":"126"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-S","rank_in_archive_order":31,"of":52,"metrics":{"FID":"2.23","NFE":"126"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"EDM2-XS","rank_in_archive_order":39,"of":52,"metrics":{"FID":"2.91","NFE":"126"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02696","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02696"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dopplerchase/cira-diff","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nvlabs/edm2","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gohyojun15/ANT_diffusion","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mmathew23/improved_edm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FutureXiang/edm2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/ema-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YichengDWu/tinyedm","reach":null}],"summary":{"ran_violates":6,"ran_draft_wrong":1,"ran_honours":1,"unverified":10},"by_repo_kind":{"listed":{"samples":13,"ran":3,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa15eaeb5d195280","entry":"default","repo":"lucidrains/ema-pytorch","repo_kind":"listed","path":"ema_pytorch/ema_pytorch.py","file_url":"https://github.com/lucidrains/ema-pytorch/blob/HEAD/ema_pytorch/ema_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa15eaeb5d195280"}},{"code_sha256_prefix":"f912c209e9c899ca","entry":"divisible_by","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f912c209e9c899ca"}},{"code_sha256_prefix":"a14db9805bf41197","entry":"edm_sampler","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a14db9805bf41197"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"bf7b2572204d3d13","entry":"first","repo":"lucidrains/ema-pytorch","repo_kind":"listed","path":"ema_pytorch/post_hoc_ema.py","file_url":"https://github.com/lucidrains/ema-pytorch/blob/HEAD/ema_pytorch/post_hoc_ema.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf7b2572204d3d13"}},{"code_sha256_prefix":"b336b517049dc932","entry":"get_default_steps","repo":"FutureXiang/edm2","repo_kind":"listed","path":"sample.py","file_url":"https://github.com/FutureXiang/edm2/blob/HEAD/sample.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b336b517049dc932"}},{"code_sha256_prefix":"cacd4f6ec202d9b4","entry":"parse_int_list","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"cacd4f6ec202d9b4"}},{"code_sha256_prefix":"4fa6b80d0288cea1","entry":"get_down_block","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fa6b80d0288cea1"}},{"code_sha256_prefix":"eade47d2a50d3683","entry":"get_down_block","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model_a.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model_a.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eade47d2a50d3683"}},{"code_sha256_prefix":"c5a7eb8a1946519a","entry":"get_up_block","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5a7eb8a1946519a"}},{"code_sha256_prefix":"3b34295d7d094429","entry":"get_up_block","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model_a.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model_a.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b34295d7d094429"}},{"code_sha256_prefix":"104550381b6dc76a","entry":"hide_diffusers_warnings","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"ema_model.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/ema_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"104550381b6dc76a"}},{"code_sha256_prefix":"a0d505b2d6c56221","entry":"pixel_norm","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model_a.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model_a.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0d505b2d6c56221"}},{"code_sha256_prefix":"892443de8bab233d","entry":"rotate2d","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"augmentation.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/augmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"892443de8bab233d"}},{"code_sha256_prefix":"bc2b81cc2d12e161","entry":"scale2d","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"augmentation.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/augmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc2b81cc2d12e161"}},{"code_sha256_prefix":"c8206975a3365ff0","entry":"translate2d","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"augmentation.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/augmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8206975a3365ff0"}},{"code_sha256_prefix":"a2beaf5faecf80b8","entry":"weight_normalize","repo":"mmathew23/improved_edm","repo_kind":"listed","path":"model.py","file_url":"https://github.com/mmathew23/improved_edm/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2beaf5faecf80b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}