{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/faster-diffusion-rethinking-the-role-of-unet","title":"Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference","arxiv_id":"2312.09608","date":"2023-12-15","proceeding":null,"authors":["Senmao Li","Taihang Hu","Joost Van de Weijer","Fahad Shahbaz Khan","Tao Liu","Linxuan Li","Shiqi Yang","Yaxing Wang","Ming-Ming Cheng","Jian Yang"],"abstract":"One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\\%$ and 24$\\%$ respectively, and DiT model sampling by 34$\\%$, while maintaining high-quality generation performance.","url_abs":"https://arxiv.org/abs/2312.09608v2","url_pdf":"https://arxiv.org/pdf/2312.09608v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"faster-diffusion-rethinking-the-role-of-unet","repo_url":"https://github.com/hutaihang/faster-diffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.09608","atlas_url":"https://app.syntology.ai/?focus=2312.09608","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.09608"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hutaihang/faster-diffusion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"23d7f44738089752","entry":"get_canny","repo":"hutaihang/faster-diffusion","repo_kind":"official","path":"utils_if.py","file_url":"https://github.com/hutaihang/faster-diffusion/blob/HEAD/utils_if.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23d7f44738089752"}},{"code_sha256_prefix":"284f982c02433d5c","entry":"get_promptls","repo":"hutaiHang/Faster-Diffusion","repo_kind":"official","path":"utils_sd.py","file_url":"https://github.com/hutaiHang/Faster-Diffusion/blob/HEAD/utils_sd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"284f982c02433d5c"}},{"code_sha256_prefix":"c2a212c016115360","entry":"load_512","repo":"hutaiHang/Faster-Diffusion","repo_kind":"official","path":"utils_sd.py","file_url":"https://github.com/hutaiHang/Faster-Diffusion/blob/HEAD/utils_sd.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2a212c016115360"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}