{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/harnessing-the-spatial-temporal-attention-of","title":"Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis","arxiv_id":"2304.03869","date":"2023-04-07","proceeding":"ICCV 2023 1","authors":["Qiucheng Wu","Yujian Liu","Handong Zhao","Trung Bui","Zhe Lin","Yang Zhang","Shiyu Chang"],"abstract":"Diffusion-based models have achieved state-of-the-art performance on text-to-image synthesis tasks. However, one critical limitation of these models is the low fidelity of generated images with respect to the text description, such as missing objects, mismatched attributes, and mislocated objects. One key reason for such inconsistencies is the inaccurate cross-attention to text in both the spatial dimension, which controls at what pixel region an object should appear, and the temporal dimension, which controls how different levels of details are added through the denoising steps. In this paper, we propose a new text-to-image algorithm that adds explicit control over spatial-temporal cross-attention in diffusion models. We first utilize a layout predictor to predict the pixel regions for objects mentioned in the text. We then impose spatial attention control by combining the attention over the entire text description and that over the local description of the particular object in the corresponding pixel region of that object. The temporal attention control is further added by allowing the combination weights to change at each denoising step, and the combination weights are optimized to ensure high fidelity between the image and the text. Experiments show that our method generates images with higher fidelity compared to diffusion-model-based baselines without fine-tuning the diffusion model. Our code is publicly available at https://github.com/UCSB-NLP-Chang/Diffusion-SpaceTime-Attn.","url_abs":"https://arxiv.org/abs/2304.03869v1","url_pdf":"https://arxiv.org/pdf/2304.03869v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"harnessing-the-spatial-temporal-attention-of","repo_url":"https://github.com/ucsb-nlp-chang/diffusion-spacetime-attn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2304.03869","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.03869"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ucsb-nlp-chang/diffusion-spacetime-attn","reach":null}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dbb6a642cadd75ab","entry":"CrossAttention","repo":"ucsb-nlp-chang/diffusion-spacetime-attn","repo_kind":"official","path":"attention_optimization/stable-diffusion/ldm/modules/attention.py","file_url":"https://github.com/ucsb-nlp-chang/diffusion-spacetime-attn/blob/HEAD/attention_optimization/stable-diffusion/ldm/modules/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbb6a642cadd75ab"}},{"code_sha256_prefix":"77fb9b023531cd68","entry":"BasicTransformerBlock","repo":"ucsb-nlp-chang/diffusion-spacetime-attn","repo_kind":"official","path":"attention_optimization/stable-diffusion/ldm/modules/attention.py","file_url":"https://github.com/ucsb-nlp-chang/diffusion-spacetime-attn/blob/HEAD/attention_optimization/stable-diffusion/ldm/modules/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77fb9b023531cd68"}},{"code_sha256_prefix":"60413c6fb508084a","entry":"SpatialTransformer","repo":"ucsb-nlp-chang/diffusion-spacetime-attn","repo_kind":"official","path":"attention_optimization/stable-diffusion/ldm/modules/attention.py","file_url":"https://github.com/ucsb-nlp-chang/diffusion-spacetime-attn/blob/HEAD/attention_optimization/stable-diffusion/ldm/modules/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60413c6fb508084a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}