{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/streamdiffusion-a-pipeline-level-solution-for","title":"StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation","arxiv_id":"2312.12491","date":"2023-12-19","proceeding":null,"authors":["Akio Kodaira","Chenfeng Xu","Toshiki Hazama","Takanori Yoshimoto","Kohei Ohno","Shogo Mitsuhori","Soichi Sugano","Hanying Cho","Zhijian Liu","Kurt Keutzer"],"abstract":"We introduce StreamDiffusion, a real-time diffusion pipeline designed for interactive image generation. Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction. This limitation becomes particularly evident in scenarios involving continuous input, such as Metaverse, live video streaming, and broadcasting, where high throughput is imperative. To address this, we present a novel approach that transforms the original sequential denoising into the batching denoising process. Stream Batch eliminates the conventional wait-and-interact approach and enables fluid and high throughput streams. To handle the frequency disparity between data input and model throughput, we design a novel input-output queue for parallelizing the streaming process. Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG), which requires additional U-Net computation. To mitigate the redundant computations, we propose a novel residual classifier-free guidance (RCFG) algorithm that reduces the number of negative conditional denoising steps to only one or even zero. Besides, we introduce a stochastic similarity filter(SSF) to optimize power consumption. Our Stream Batch achieves around 1.5x speedup compared to the sequential denoising method at different denoising levels. The proposed RCFG leads to speeds up to 2.05x higher than the conventional CFG. Combining the proposed strategies and existing mature acceleration tools makes the image-to-image generation achieve up-to 91.07fps on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers over 59.56x. Furthermore, our proposed StreamDiffusion also significantly reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one RTX4090, respectively.","url_abs":"https://arxiv.org/abs/2312.12491v1","url_pdf":"https://arxiv.org/pdf/2312.12491v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"streamdiffusion-a-pipeline-level-solution-for","repo_url":"https://github.com/cumulo-autumn/streamdiffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"u-net","method_name":"U-Net"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.12491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.12491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cumulo-autumn/streamdiffusion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d1f7607f85bbbcf6","entry":"create_onnx_path","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/acceleration/tensorrt/builder.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/acceleration/tensorrt/builder.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1f7607f85bbbcf6"}},{"code_sha256_prefix":"16ce98a3e3130938","entry":"decode_images","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/acceleration/tensorrt/utilities.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/acceleration/tensorrt/utilities.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16ce98a3e3130938"}},{"code_sha256_prefix":"5dd5ccc011afb9c3","entry":"denormalize","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/image_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/image_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dd5ccc011afb9c3"}},{"code_sha256_prefix":"35d471d9e81ee42a","entry":"is_installed","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/pip_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/pip_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35d471d9e81ee42a"}},{"code_sha256_prefix":"b188597984f52631","entry":"numpy_to_pil","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/image_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/image_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b188597984f52631"}},{"code_sha256_prefix":"7f011624871aea53","entry":"preprocess_image","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/acceleration/tensorrt/utilities.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/acceleration/tensorrt/utilities.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f011624871aea53"}},{"code_sha256_prefix":"e0ca6f492a8f2cbf","entry":"pt_to_numpy","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/image_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/image_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0ca6f492a8f2cbf"}},{"code_sha256_prefix":"635ea667a5c67443","entry":"version","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/pip_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/pip_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"635ea667a5c67443"}},{"code_sha256_prefix":"ef16b546127f1484","entry":"run_python","repo":"cumulo-autumn/streamdiffusion","repo_kind":"official","path":"src/streamdiffusion/pip_utils.py","file_url":"https://github.com/cumulo-autumn/streamdiffusion/blob/HEAD/src/streamdiffusion/pip_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef16b546127f1484"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}