{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-generate-semantic-layouts-for","title":"Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis","arxiv_id":"2308.08157","date":"2023-08-16","proceeding":"ICCV 2023 1","authors":["Minho Park","Jooyeol Yun","Seunghwan Choi","Jaegul Choo"],"abstract":"Existing text-to-image generation approaches have set high standards for photorealism and text-image correspondence, largely benefiting from web-scale text-image datasets, which can include up to 5~billion pairs. However, text-to-image generation models trained on domain-specific datasets, such as urban scenes, medical images, and faces, still suffer from low text-image correspondence due to the lack of text-image pairs. Additionally, collecting billions of text-image pairs for a specific domain can be time-consuming and costly. Thus, ensuring high text-image correspondence without relying on web-scale text-image datasets remains a challenging task. In this paper, we present a novel approach for enhancing text-image correspondence by leveraging available semantic layouts. Specifically, we propose a Gaussian-categorical diffusion process that simultaneously generates both images and corresponding layout pairs. Our experiments reveal that we can guide text-to-image generation models to be aware of the semantics of different image regions, by training the model to generate semantic labels for each pixel. We demonstrate that our approach achieves higher text-image correspondence compared to existing text-to-image generation approaches in the Multi-Modal CelebA-HQ and the Cityscapes dataset, where text-image pairs are scarce. Codes are available in this https://pmh9960.github.io/research/GCDP","url_abs":"https://arxiv.org/abs/2308.08157v1","url_pdf":"https://arxiv.org/pdf/2308.08157v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-generate-semantic-layouts-for","repo_url":"https://github.com/pmh9960/GCDP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"multimodal-generation","task_name":"multimodal generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.08157","atlas_url":"https://app.syntology.ai/?focus=2308.08157","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.08157"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pmh9960/GCDP","reach":{"status":"ok"}}],"summary":{"ran_honours":4,"ran_draft_wrong":1,"ran":6,"ran_violates":3,"unverified":3},"by_repo_kind":{"official":{"samples":17,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"7f1040f5e3991d5e","entry":"identity","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/imagen_pytorch.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/imagen_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f1040f5e3991d5e"}},{"code_sha256_prefix":"b5472673332de78e","entry":"log","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/elucidated_imagen.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/elucidated_imagen.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5472673332de78e"}},{"code_sha256_prefix":"fc8652b9a9069275","entry":"ListOrTuple","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/configs.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc8652b9a9069275"}},{"code_sha256_prefix":"efcd80d66cfe5bcb","entry":"cast_tuple","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/trainer.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efcd80d66cfe5bcb"}},{"code_sha256_prefix":"a47706b35198fab3","entry":"convert_image_to","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/data.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a47706b35198fab3"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/configs.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/configs.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/configs.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/configs.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"3072294c9b32cf4b","entry":"first","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/imagen_pytorch.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/imagen_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3072294c9b32cf4b"}},{"code_sha256_prefix":"0ba8f745cf192018","entry":"get_images_dataloader","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/data.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ba8f745cf192018"}},{"code_sha256_prefix":"479771b269fc5883","entry":"log_1_min_a","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/joint_imagen.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/joint_imagen.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"479771b269fc5883"}},{"code_sha256_prefix":"032c0dece6594020","entry":"log_add_exp","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/joint_imagen.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/joint_imagen.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"032c0dece6594020"}},{"code_sha256_prefix":"a1229a9f4df127e2","entry":"normalize_to_neg_one_to_one","repo":"pmh9960/GCDP","repo_kind":"official","path":"datasets/celeba.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/datasets/celeba.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1229a9f4df127e2"}},{"code_sha256_prefix":"42a39b682ff0811f","entry":"safeget","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/utils.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a39b682ff0811f"}},{"code_sha256_prefix":"532db2cf579e3b08","entry":"unnormalize_and_clamp_to_zero_to_one","repo":"pmh9960/GCDP","repo_kind":"official","path":"datasets/celeba.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/datasets/celeba.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"532db2cf579e3b08"}},{"code_sha256_prefix":"09c8479d9a5b3e06","entry":"extract","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/joint_imagen.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/joint_imagen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09c8479d9a5b3e06"}},{"code_sha256_prefix":"4af30ae5e0b7d6ca","entry":"get_tokenizer","repo":"pmh9960/GCDP","repo_kind":"official","path":"imagen_pytorch/t5.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/imagen_pytorch/t5.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4af30ae5e0b7d6ca"}},{"code_sha256_prefix":"6e53d3fe76720ef6","entry":"unnormalize_to_zero_to_one","repo":"pmh9960/GCDP","repo_kind":"official","path":"datasets/celeba.py","file_url":"https://github.com/pmh9960/GCDP/blob/HEAD/datasets/celeba.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e53d3fe76720ef6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}