{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-diffusion-transformers-to-16-billion","title":"Scaling Diffusion Transformers to 16 Billion Parameters","arxiv_id":"2407.11633","date":"2024-07-16","proceeding":null,"authors":["Zhengcong Fei","Mingyuan Fan","Changqian Yu","Debang Li","Junshi Huang"],"abstract":"In this paper, we present DiT-MoE, a sparse version of the diffusion Transformer, that is scalable and competitive with dense networks while exhibiting highly optimized inference. The DiT-MoE includes two simple designs: shared expert routing and expert-level balance loss, thereby capturing common knowledge and reducing redundancy among the different routed experts. When applied to conditional image generation, a deep analysis of experts specialization gains some interesting observations: (i) Expert selection shows preference with spatial position and denoising time step, while insensitive with different class-conditional information; (ii) As the MoE layers go deeper, the selection of experts gradually shifts from specific spacial position to dispersion and balance. (iii) Expert specialization tends to be more concentrated at the early time step and then gradually uniform after half. We attribute it to the diffusion process that first models the low-frequency spatial information and then high-frequency complex information. Based on the above guidance, a series of DiT-MoE experimentally achieves performance on par with dense networks yet requires much less computational load during inference. More encouragingly, we demonstrate the potential of DiT-MoE with synthesized image data, scaling diffusion model at a 16.5B parameter that attains a new SoTA FID-50K score of 1.80 in 512$\\times$512 resolution settings. The project page: https://github.com/feizc/DiT-MoE.","url_abs":"https://arxiv.org/abs/2407.11633v3","url_pdf":"https://arxiv.org/pdf/2407.11633v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-diffusion-transformers-to-16-billion","repo_url":"https://github.com/feizc/dit-moe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"conditional-image-generation","task_name":"Conditional Image Generation"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":null,"task_name":"Position"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"moe","method_name":"MoE"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.11633","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.11633"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/feizc/dit-moe","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":6,"ran_honours":4,"ran_violates":1,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"d6a68e210556f857","entry":"approx_standard_normal_cdf","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/diffusion_utils.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/diffusion_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6a68e210556f857"}},{"code_sha256_prefix":"1712a07966b542ee","entry":"center_crop_arr","repo":"feizc/dit-moe","repo_kind":"official","path":"train_deepspeed.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/train_deepspeed.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1712a07966b542ee"}},{"code_sha256_prefix":"ab1c9568b4e13899","entry":"continuous_gaussian_log_likelihood","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/diffusion_utils.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/diffusion_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab1c9568b4e13899"}},{"code_sha256_prefix":"7b21a01ae77703a3","entry":"create_npz_from_sample_folder","repo":"feizc/dit-moe","repo_kind":"official","path":"sample_ddp.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/sample_ddp.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b21a01ae77703a3"}},{"code_sha256_prefix":"c92c27c924b517e8","entry":"get_2d_sincos_pos_embed","repo":"feizc/dit-moe","repo_kind":"official","path":"models.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/models.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c92c27c924b517e8"}},{"code_sha256_prefix":"3e0fa4efc22272d4","entry":"get_beta_schedule","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/gaussian_diffusion.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e0fa4efc22272d4"}},{"code_sha256_prefix":"36e30c7fb679ec78","entry":"get_named_beta_schedule","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/gaussian_diffusion.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36e30c7fb679ec78"}},{"code_sha256_prefix":"f6d7c009a8efb8b7","entry":"mean_flat","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/gaussian_diffusion.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6d7c009a8efb8b7"}},{"code_sha256_prefix":"03310bba324ae4fb","entry":"modulate","repo":"feizc/dit-moe","repo_kind":"official","path":"models.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/models.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03310bba324ae4fb"}},{"code_sha256_prefix":"8afbfc42c6ea0448","entry":"normal_kl","repo":"feizc/dit-moe","repo_kind":"official","path":"diffusion/diffusion_utils.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/diffusion/diffusion_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8afbfc42c6ea0448"}},{"code_sha256_prefix":"86c7147edb17dd5d","entry":"normalization","repo":"feizc/dit-moe","repo_kind":"official","path":"analysis/heatmap_class.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/analysis/heatmap_class.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86c7147edb17dd5d"}},{"code_sha256_prefix":"b789e06f5968456b","entry":"static_for_class","repo":"feizc/dit-moe","repo_kind":"official","path":"analysis/heatmap_class.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/analysis/heatmap_class.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b789e06f5968456b"}},{"code_sha256_prefix":"8382f7e48cc8a309","entry":"create_logger","repo":"feizc/dit-moe","repo_kind":"official","path":"train_deepspeed.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/train_deepspeed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8382f7e48cc8a309"}},{"code_sha256_prefix":"6a0d5ecd441905cc","entry":"download_model","repo":"feizc/dit-moe","repo_kind":"official","path":"download.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/download.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a0d5ecd441905cc"}},{"code_sha256_prefix":"e8e07fe599aa424e","entry":"find_model","repo":"feizc/dit-moe","repo_kind":"official","path":"download.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/download.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8e07fe599aa424e"}},{"code_sha256_prefix":"665d8a4e8f673a4c","entry":"get_2d_sincos_pos_embed_from_grid","repo":"feizc/dit-moe","repo_kind":"official","path":"models.py","file_url":"https://github.com/feizc/dit-moe/blob/HEAD/models.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"665d8a4e8f673a4c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}