{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-with-forward-models-solving","title":"Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision","arxiv_id":"2306.11719","date":"2023-06-20","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Denoising diffusion models are a powerful type of generative models used to capture complex distributions of real-world signals. However, their applicability is limited to scenarios where training samples are readily available, which is not always the case in real-world applications. For example, in inverse graphics, the goal is to generate samples from a distribution of 3D scenes that align with a given image, but ground-truth 3D scenes are unavailable and only 2D images are accessible. To address this limitation, we propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. Instead, these signals are measured indirectly through a known differentiable forward model, which produces partial observations of the unknown signal. Our approach involves integrating the forward model directly into the denoising process. This integration effectively connects the generative modeling of observations with the generative modeling of the underlying signals, allowing for end-to-end training of a conditional generative model over signals. During inference, our approach enables sampling from the distribution of underlying signals that are consistent with a given partial observation. We demonstrate the effectiveness of our method on three challenging computer vision tasks. For instance, in the context of inverse graphics, our model enables direct sampling from the distribution of 3D scenes that align with a single 2D input image.","url_abs":"https://arxiv.org/abs/2306.11719v2","url_pdf":"https://arxiv.org/pdf/2306.11719v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-with-forward-models-solving","repo_url":"https://github.com/ayushtewari/DFM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.11719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.11719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ayushtewari/DFM","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"487b39c92b4aed7c","entry":"get_train_settings","repo":"ayushtewari/DFM","repo_kind":"listed","path":"experiment_scripts/train_3D_diffusion.py","file_url":"https://github.com/ayushtewari/DFM/blob/HEAD/experiment_scripts/train_3D_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"487b39c92b4aed7c"}},{"code_sha256_prefix":"954d123f430c091c","entry":"prepare_depth","repo":"ayushtewari/DFM","repo_kind":"listed","path":"experiment_scripts/co3d_results.py","file_url":"https://github.com/ayushtewari/DFM/blob/HEAD/experiment_scripts/co3d_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"954d123f430c091c"}},{"code_sha256_prefix":"513ea8d111a71728","entry":"prepare_video_out","repo":"ayushtewari/DFM","repo_kind":"listed","path":"experiment_scripts/co3d_results.py","file_url":"https://github.com/ayushtewari/DFM/blob/HEAD/experiment_scripts/co3d_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"513ea8d111a71728"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}