{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-spatially-sparse-inference-for","title":"Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models","arxiv_id":"2211.02048","date":"2022-11-03","proceeding":null,"authors":["Muyang Li","Ji Lin","Chenlin Meng","Stefano Ermon","Song Han","Jun-Yan Zhu"],"abstract":"During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models. Our key observation is that users prone to gradually edit the input image. This motivates us to cache and reuse the feature maps of the original image. Given an edited image, we sparsely apply the convolutional filters to the edited regions while reusing the cached features for the unedited areas. Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on off-the-shelf hardware. With about $1\\%$-area edits, SIGE accelerates DDPM by $3.0\\times$ on NVIDIA RTX 3090 and $4.6\\times$ on Apple M1 Pro GPU, Stable Diffusion by $7.2\\times$ on 3090, and GauGAN by $5.6\\times$ on 3090 and $5.2\\times$ on M1 Pro GPU. Compared to our conference version, we extend SIGE to accommodate attention layers and apply it to Stable Diffusion. Additionally, we offer support for Apple M1 Pro GPU and include more results with large and sequential edits.","url_abs":"https://arxiv.org/abs/2211.02048v4","url_pdf":"https://arxiv.org/pdf/2211.02048v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-spatially-sparse-inference-for","repo_url":"https://github.com/lmxyy/sige","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.02048","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.02048"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/lmxyy/sige","reach":null}],"summary":{"ran":3,"ran_fixture":1,"unverified":13},"by_repo_kind":{"official":{"samples":17,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"7a0f92aace8379bd","entry":"Downsample","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7a0f92aace8379bd"}},{"code_sha256_prefix":"335c75661060896d","entry":"SIGEConv2d","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"335c75661060896d"}},{"code_sha256_prefix":"d507545007f66e19","entry":"SIGEModule","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d507545007f66e19"}},{"code_sha256_prefix":"b8dedf14898e5502","entry":"get_timestep_embedding","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b8dedf14898e5502"}},{"code_sha256_prefix":"e6819730eb3cbd0f","entry":"Gather","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e6819730eb3cbd0f"}},{"code_sha256_prefix":"5fd30a8dc69ba054","entry":"SIGEDownsample","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5fd30a8dc69ba054"}},{"code_sha256_prefix":"ab08502e82055fad","entry":"SIGEFusedAttnBlock","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ab08502e82055fad"}},{"code_sha256_prefix":"4f54fe4195f44e4a","entry":"SIGEFusedResnetBlock","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4f54fe4195f44e4a"}},{"code_sha256_prefix":"045129cab1cb8c3a","entry":"SIGEFusedUNet","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"045129cab1cb8c3a"}},{"code_sha256_prefix":"024f482cc587c98d","entry":"SIGEModel","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"024f482cc587c98d"}},{"code_sha256_prefix":"8a575c88399bbcf7","entry":"SIGEModuleWrapper","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8a575c88399bbcf7"}},{"code_sha256_prefix":"f153297ee40e8b8d","entry":"SIGEUpsample","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f153297ee40e8b8d"}},{"code_sha256_prefix":"aae8fcf6a37d95da","entry":"Scatter","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"aae8fcf6a37d95da"}},{"code_sha256_prefix":"c3a4f27f51286763","entry":"ScatterGather","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c3a4f27f51286763"}},{"code_sha256_prefix":"198eb290f706c5a3","entry":"ScatterWithBlockResidual","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"198eb290f706c5a3"}},{"code_sha256_prefix":"1dc30e6b525d42ae","entry":"my_group_norm","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1dc30e6b525d42ae"}},{"code_sha256_prefix":"153647897a024a30","entry":"reduce_mask","repo":"lmxyy/sige","repo_kind":"official","path":"diffusion/models/ddpm_arch/sige_fused_unet.py","file_url":"https://github.com/lmxyy/sige/blob/HEAD/diffusion/models/ddpm_arch/sige_fused_unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"153647897a024a30"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}