{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/layoutllm-t2i-eliciting-layout-guidance-from","title":"LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation","arxiv_id":"2308.05095","date":"2023-08-09","proceeding":null,"authors":["Leigang Qu","Shengqiong Wu","Hao Fei","Liqiang Nie","Tat-Seng Chua"],"abstract":"In the text-to-image generation field, recent remarkable progress in Stable Diffusion makes it possible to generate rich kinds of novel photorealistic images. However, current models still face misalignment issues (e.g., problematic spatial relation understanding and numeration failure) in complex natural scenes, which impedes the high-faithfulness text-to-image generation. Although recent efforts have been made to improve controllability by giving fine-grained guidance (e.g., sketch and scribbles), this issue has not been fundamentally tackled since users have to provide such guidance information manually. In this work, we strive to synthesize high-fidelity images that are semantically aligned with a given textual prompt without any guidance. Toward this end, we propose a coarse-to-fine paradigm to achieve layout planning and image generation. Concretely, we first generate the coarse-grained layout conditioned on a given textual prompt via in-context learning based on Large Language Models. Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art models in terms of layout and image generation. Our code and settings are available at https://layoutllm-t2i.github.io.","url_abs":"https://arxiv.org/abs/2308.05095v2","url_pdf":"https://arxiv.org/pdf/2308.05095v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"layoutllm-t2i-eliciting-layout-guidance-from","repo_url":"https://github.com/layoutllm-t2i/layoutllm-t2i","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.05095","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.05095"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/layoutllm-t2i/layoutllm-t2i","reach":null}],"summary":{"ran":7,"ran_fixture":1,"ran_honours":2,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"listed":{"samples":17,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"62f5f6f02082d074","entry":"CheckpointFunction","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62f5f6f02082d074"}},{"code_sha256_prefix":"62dab2e5d80c65f2","entry":"CrossAttention","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62dab2e5d80c65f2"}},{"code_sha256_prefix":"685ba45c824392d7","entry":"GatedCrossAttentionDense","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"685ba45c824392d7"}},{"code_sha256_prefix":"3c992fc8cf1bcb11","entry":"GatedSelfAttentionDense","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c992fc8cf1bcb11"}},{"code_sha256_prefix":"6fda5be5b47ceae2","entry":"GatedSelfAttentionDense2","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fda5be5b47ceae2"}},{"code_sha256_prefix":"78f4c181439d2228","entry":"RelationCrossAttention","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78f4c181439d2228"}},{"code_sha256_prefix":"35a9d16404c55eee","entry":"SelfAttention","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35a9d16404c55eee"}},{"code_sha256_prefix":"d123d0ead32f6b21","entry":"alpha_generator","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"txt2img.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/txt2img.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d123d0ead32f6b21"}},{"code_sha256_prefix":"d0408ce6cb5db238","entry":"checkpoint","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0408ce6cb5db238"}},{"code_sha256_prefix":"519518427f6e4e2f","entry":"project","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"txt2img.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/txt2img.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"519518427f6e4e2f"}},{"code_sha256_prefix":"6ffddab9b4a35318","entry":"resume","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"train_rl.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/train_rl.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ffddab9b4a35318"}},{"code_sha256_prefix":"71d199572f378540","entry":"BasicTransformerBlock","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71d199572f378540"}},{"code_sha256_prefix":"1ce47bfbc5b62bab","entry":"GligenCombineLayout","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ce47bfbc5b62bab"}},{"code_sha256_prefix":"b79532254b752520","entry":"ResBlock","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b79532254b752520"}},{"code_sha256_prefix":"3f6fea66d029ae8a","entry":"SpatialTransformer","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f6fea66d029ae8a"}},{"code_sha256_prefix":"8bab56c818b37ac8","entry":"TimestepEmbedSequential","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bab56c818b37ac8"}},{"code_sha256_prefix":"f0258fd515fdbbe6","entry":"UNetModel","repo":"layoutllm-t2i/layoutllm-t2i","repo_kind":"listed","path":"GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","file_url":"https://github.com/layoutllm-t2i/layoutllm-t2i/blob/HEAD/GLIGEN/ldm/modules/diffusionmodules/gligen_combine_layout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0258fd515fdbbe6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}