{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generating-diverse-structure-for-image","title":"Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE","arxiv_id":"2103.10022","date":"2021-03-18","proceeding":"CVPR 2021 1","authors":["Jialun Peng","Dong Liu","Songcen Xu","Houqiang Li"],"abstract":"Given an incomplete image without additional constraint, image inpainting natively allows for multiple solutions as long as they appear plausible. Recently, multiplesolution inpainting methods have been proposed and shown the potential of generating diverse results. However, these methods have difficulty in ensuring the quality of each solution, e.g. they produce distorted structure and/or blurry texture. We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. The proposed model is inspired by the hierarchical vector quantized variational auto-encoder (VQ-VAE), whose hierarchical architecture isentangles structural and textural information. In addition, the vector quantization in VQVAE enables autoregressive modeling of the discrete distribution over the structural information. Sampling from the distribution can easily generate diverse and high-quality structures, making up the first stage of our model. In the second stage, we propose a structural attention module inside the texture generation network, where the module utilizes the structural information to capture distant correlations. We further reuse the VQ-VAE to calculate two feature losses, which help improve structure coherence and texture realism, respectively. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images. Code and models are available at: https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting.","url_abs":"https://arxiv.org/abs/2103.10022v1","url_pdf":"https://arxiv.org/pdf/2103.10022v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generating-diverse-structure-for-image","repo_url":"https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"generating-diverse-structure-for-image","repo_url":"https://github.com/wuyx517/VQVAE_image_inpainting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-inpainting","task_name":"Image Inpainting"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"texture-synthesis","task_name":"Texture Synthesis"}],"methods":[{"method_slug":"pixel-prediction","method_name":"Inpainting"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2103.10022","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.10022"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wuyx517/VQVAE_image_inpainting","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting","reach":{"status":"unanswered"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"listed":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"a4a78f443b2b7133","entry":"Decoder","repo":"wuyx517/VQVAE_image_inpainting","repo_kind":"listed","path":"model/vqvae.py","file_url":"https://github.com/wuyx517/VQVAE_image_inpainting/blob/HEAD/model/vqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4a78f443b2b7133"}},{"code_sha256_prefix":"52f9bcecdbf34490","entry":"Encoder","repo":"wuyx517/VQVAE_image_inpainting","repo_kind":"listed","path":"model/vqvae.py","file_url":"https://github.com/wuyx517/VQVAE_image_inpainting/blob/HEAD/model/vqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52f9bcecdbf34490"}},{"code_sha256_prefix":"8afc7838ff02a027","entry":"Resnet","repo":"wuyx517/VQVAE_image_inpainting","repo_kind":"listed","path":"model/vqvae.py","file_url":"https://github.com/wuyx517/VQVAE_image_inpainting/blob/HEAD/model/vqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8afc7838ff02a027"}},{"code_sha256_prefix":"4bfddced8570d768","entry":"VectorQuantizer","repo":"wuyx517/VQVAE_image_inpainting","repo_kind":"listed","path":"model/vqvae.py","file_url":"https://github.com/wuyx517/VQVAE_image_inpainting/blob/HEAD/model/vqvae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bfddced8570d768"}},{"code_sha256_prefix":"5c4d71e019c2e7eb","entry":"VQVAE","repo":"wuyx517/VQVAE_image_inpainting","repo_kind":"listed","path":"model/vqvae.py","file_url":"https://github.com/wuyx517/VQVAE_image_inpainting/blob/HEAD/model/vqvae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c4d71e019c2e7eb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}