{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ddt-decoupled-diffusion-transformer-1","title":"DDT: Decoupled Diffusion Transformer","arxiv_id":"2504.05741","date":"2025-04-08","proceeding":null,"authors":["Shuai Wang","Zhi Tian","Weilin Huang","LiMin Wang"],"abstract":"Diffusion transformers have demonstrated remarkable generation quality, albeit requiring longer training iterations and numerous inference steps. In each denoising step, diffusion transformers encode the noisy inputs to extract the lower-frequency semantic component and then decode the higher frequency with identical modules. This scheme creates an inherent optimization dilemma: encoding low-frequency semantics necessitates reducing high-frequency components, creating tension between semantic encoding and high-frequency decoding. To resolve this challenge, we propose a new \\textbf{\\color{ddt}D}ecoupled \\textbf{\\color{ddt}D}iffusion \\textbf{\\color{ddt}T}ransformer~(\\textbf{\\color{ddt}DDT}), with a decoupled design of a dedicated condition encoder for semantic extraction alongside a specialized velocity decoder. Our experiments reveal that a more substantial encoder yields performance improvements as model size increases. For ImageNet $256\\times256$, Our DDT-XL/2 achieves a new state-of-the-art performance of {1.31 FID}~(nearly $4\\times$ faster training convergence compared to previous diffusion transformers). For ImageNet $512\\times512$, Our DDT-XL/2 achieves a new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our decoupled architecture enhances inference speed by enabling the sharing self-condition between adjacent denoising steps. To minimize performance degradation, we propose a novel statistical dynamic programming approach to identify optimal sharing strategies.","url_abs":"https://arxiv.org/abs/2504.05741v2","url_pdf":"https://arxiv.org/pdf/2504.05741v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ddt-decoupled-diffusion-transformer-1","repo_url":"https://github.com/MCG-NJU/DDT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"DDT-XL/2(22en6de 675M + guidance interval )","rank_in_archive_order":8,"of":94,"metrics":{"FID":"1.26","Inception score":"310.6"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"DDT-XL/2(22en6de 675M + guidance interval )","rank_in_archive_order":5,"of":52,"metrics":{"FID":"1.28","Inception score":"305","NFE":"500"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.05741","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}