{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/structure-matters-tackling-the-semantic","title":"Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting","arxiv_id":"2403.19898","date":"2024-03-29","proceeding":"CVPR 2024 1","authors":["Haipeng Liu","Yang Wang","Biao Qian","Meng Wang","Yong Rui"],"abstract":"Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising process.Despite the meaningful semantics generation,the existing arts suffer from the semantic discrepancy between masked and unmasked regions, since the semantically dense unmasked texture fails to be completely degraded while the masked regions turn to the pure noise in diffusion process,leading to the large discrepancy between them. In this paper,we aim to answer how unmasked semantics guide texture denoising process;together with how to tackle the semantic discrepancy,to facilitate the consistent and meaningful semantics generation. To this end,we propose a novel structure-guided diffusion model named StrDiffusion,to reformulate the conventional texture denoising process under structure guidance to derive a simplified denoising objective for image inpainting,while revealing:1)the semantically sparse structure is beneficial to tackle semantic discrepancy in early stage, while dense texture generates reasonable semantics in late stage;2)the semantics from unmasked regions essentially offer the time-dependent structure guidance for the texture denoising process,benefiting from the time-dependent sparsity of the structure semantics.For the denoising process,a structure-guided neural network is trained to estimate the simplified denoising objective by exploiting the consistency of the denoised structure between masked and unmasked regions.Besides,we devise an adaptive resampling strategy as a formal criterion as whether structure is competent to guide the texture denoising process,while regulate their semantic correlations.Extensive experiments validate the merits of StrDiffusion over the state-of-the-arts.Our code is available at https://github.com/htyjers/StrDiffusion.","url_abs":"https://arxiv.org/abs/2403.19898v2","url_pdf":"https://arxiv.org/pdf/2403.19898v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"structure-matters-tackling-the-semantic","repo_url":"https://github.com/htyjers/strdiffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-inpainting","task_name":"Image Inpainting"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"pixel-prediction","method_name":"Inpainting"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.19898","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.19898"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/htyjers/strdiffusion","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/htyjers/StrDiffusion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":10},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"40c52ad98161c17b","entry":"dict2str","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/options.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/options.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40c52ad98161c17b"}},{"code_sha256_prefix":"ada7273eee7081c7","entry":"dict_to_nonedict","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/options.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/options.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ada7273eee7081c7"}},{"code_sha256_prefix":"3ac11bfe0ae80195","entry":"variance_scaling","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/layers.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/layers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac11bfe0ae80195"}},{"code_sha256_prefix":"9f13fd20494eac47","entry":"dense","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/dense_layer.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/dense_layer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f13fd20494eac47"}},{"code_sha256_prefix":"db5f935a1d6b4c2c","entry":"get_act","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/layers.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db5f935a1d6b4c2c"}},{"code_sha256_prefix":"8db300b9f1afb2d3","entry":"get_weight","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/up_or_down_sampling.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/up_or_down_sampling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8db300b9f1afb2d3"}},{"code_sha256_prefix":"f364b5647cd1f451","entry":"gray_to_edge","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/canny.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/canny.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f364b5647cd1f451"}},{"code_sha256_prefix":"b42da332ce0ead42","entry":"kaiming_uniform_","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/dense_layer.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/dense_layer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b42da332ce0ead42"}},{"code_sha256_prefix":"e0ca74cc0ccd0e50","entry":"naive_downsample_2d","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/up_or_down_sampling.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/up_or_down_sampling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0ca74cc0ccd0e50"}},{"code_sha256_prefix":"f425357463a3de36","entry":"naive_upsample_2d","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/up_or_down_sampling.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/up_or_down_sampling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f425357463a3de36"}},{"code_sha256_prefix":"85b2235eb7afaa4f","entry":"ncsn_conv1x1","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/layers.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85b2235eb7afaa4f"}},{"code_sha256_prefix":"2178a45bccb10af5","entry":"parse","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/options.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/options.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2178a45bccb10af5"}},{"code_sha256_prefix":"49cf191f770c146a","entry":"variance_scaling_init_","repo":"htyjers/StrDiffusion","repo_kind":"official","path":"train/discriminator/config/inpainting/models/dense_layer.py","file_url":"https://github.com/htyjers/StrDiffusion/blob/HEAD/train/discriminator/config/inpainting/models/dense_layer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49cf191f770c146a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}