{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/image-inpainting-via-tractable-steering-of","title":"Image Inpainting via Tractable Steering of Diffusion Models","arxiv_id":"2401.03349","date":"2023-11-28","proceeding":null,"authors":["Anji Liu","Mathias Niepert","Guy Van Den Broeck"],"abstract":"Diffusion models are the current state of the art for generating photorealistic images. Controlling the sampling process for constrained image generation tasks such as inpainting, however, remains challenging since exact conditioning on such constraints is intractable. While existing methods use various techniques to approximate the constrained posterior, this paper proposes to exploit the ability of Tractable Probabilistic Models (TPMs) to exactly and efficiently compute the constrained posterior, and to leverage this signal to steer the denoising process of diffusion models. Specifically, this paper adopts a class of expressive TPMs termed Probabilistic Circuits (PCs). Building upon prior advances, we further scale up PCs and make them capable of guiding the image generation process of diffusion models. Empirical results suggest that our approach can consistently improve the overall quality and semantic coherence of inpainted images across three natural image datasets (i.e., CelebA-HQ, ImageNet, and LSUN) with only $\\sim\\! 10 \\%$ additional computational overhead brought by the TPM. Further, with the help of an image encoder and decoder, our method can readily accept semantic constraints on specific regions of the image, which opens up the potential for more controlled image generation tasks. In addition to proposing a new framework for constrained image generation, this paper highlights the benefit of more tractable models and motivates the development of expressive TPMs.","url_abs":"https://arxiv.org/abs/2401.03349v2","url_pdf":"https://arxiv.org/pdf/2401.03349v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"image-inpainting-via-tractable-steering-of","repo_url":"https://github.com/ucla-starai/tiramisu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-inpainting","task_name":"Image Inpainting"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.03349","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.03349"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ucla-starai/tiramisu","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_honours":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"732fba0548da35e1","entry":"construct_divisions","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/img_based_structures/shared_pd.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/img_based_structures/shared_pd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"732fba0548da35e1"}},{"code_sha256_prefix":"de254adf0261a4e3","entry":"get_activation","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/latent_transformer_2d/attention.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/latent_transformer_2d/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de254adf0261a4e3"}},{"code_sha256_prefix":"d3fcbed7ad4cbcae","entry":"get_prod_lls","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/lvd/apply_lvd.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/lvd/apply_lvd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3fcbed7ad4cbcae"}},{"code_sha256_prefix":"479771b269fc5883","entry":"log_1_min_a","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"479771b269fc5883"}},{"code_sha256_prefix":"032c0dece6594020","entry":"log_add_exp","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"032c0dece6594020"}},{"code_sha256_prefix":"5eb945b1f970f6af","entry":"sum_except_batch","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/latent_transformer_2d/diffusion_transformer.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5eb945b1f970f6af"}},{"code_sha256_prefix":"ec6863a54e565926","entry":"get_sum_lls","repo":"ucla-starai/tiramisu","repo_kind":"official","path":"controlled_img_modeling/lvd/apply_lvd.py","file_url":"https://github.com/ucla-starai/tiramisu/blob/HEAD/controlled_img_modeling/lvd/apply_lvd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec6863a54e565926"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}