{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ddfm-denoising-diffusion-model-for-multi","title":"DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion","arxiv_id":"2303.06840","date":"2023-03-13","proceeding":"ICCV 2023 1","authors":["Zixiang Zhao","Haowen Bai","Yuanzhi Zhu","Jiangshe Zhang","Shuang Xu","Yulun Zhang","Kai Zhang","Deyu Meng","Radu Timofte","Luc van Gool"],"abstract":"Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at \\url{https://github.com/Zhaozixiang1228/MMIF-DDFM}.","url_abs":"https://arxiv.org/abs/2303.06840v2","url_pdf":"https://arxiv.org/pdf/2303.06840v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ddfm-denoising-diffusion-model-for-multi","repo_url":"https://github.com/zhaozixiang1228/mmif-ddfm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ddfm-denoising-diffusion-model-for-multi","repo_url":"https://github.com/zhaozixiang1228/if-film","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ddfm-denoising-diffusion-model-for-multi","repo_url":"https://github.com/zhaozixiang1228/mmif-cddfuse","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ddfm-denoising-diffusion-model-for-multi","repo_url":"https://github.com/zhaozixiang1228/mmif-emma","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.06840","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.06840"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaozixiang1228/mmif-emma","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaozixiang1228/if-film","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaozixiang1228/mmif-cddfuse","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaozixiang1228/mmif-ddfm","reach":null}],"summary":{"ran":15,"ran_draft_wrong":1,"ran_fixture":2,"unverified":9},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1},"listed":{"samples":21,"ran":15,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":27,"samples":[{"code_sha256_prefix":"98e962e3944f0fb6","entry":"Attention","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98e962e3944f0fb6"}},{"code_sha256_prefix":"f68c09a19fb1adc5","entry":"AttentionBase","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f68c09a19fb1adc5"}},{"code_sha256_prefix":"553a596819206f47","entry":"CrossAttention","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"553a596819206f47"}},{"code_sha256_prefix":"4eb73bbf45be6872","entry":"DetailFeatureExtraction","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4eb73bbf45be6872"}},{"code_sha256_prefix":"a4d094c84b651498","entry":"DetailNode","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4d094c84b651498"}},{"code_sha256_prefix":"7f035f55e2f08711","entry":"EM_Initial","repo":"zhaozixiang1228/mmif-ddfm","repo_kind":"official","path":"guided_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhaozixiang1228/mmif-ddfm/blob/HEAD/guided_diffusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f035f55e2f08711"}},{"code_sha256_prefix":"ef9dcfa1656c7a2d","entry":"GlobalFeatureExtraction","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef9dcfa1656c7a2d"}},{"code_sha256_prefix":"6a741c9cce81776b","entry":"InvertedResidualBlock","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a741c9cce81776b"}},{"code_sha256_prefix":"7b4409a2654eb25d","entry":"LayerNorm","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b4409a2654eb25d"}},{"code_sha256_prefix":"47b15bbe9ab60e4b","entry":"LocalFeatureExtraction","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47b15bbe9ab60e4b"}},{"code_sha256_prefix":"d41e29922999d7fa","entry":"Mlp","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d41e29922999d7fa"}},{"code_sha256_prefix":"dc2b3fa6c2ac9333","entry":"Mlp","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc2b3fa6c2ac9333"}},{"code_sha256_prefix":"5efba677a3ddb846","entry":"ResBlock","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5efba677a3ddb846"}},{"code_sha256_prefix":"7fbefc76d724c861","entry":"TransformerBlock","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fbefc76d724c861"}},{"code_sha256_prefix":"500976c0521f9f77","entry":"extract_and_expand","repo":"zhaozixiang1228/mmif-ddfm","repo_kind":"official","path":"guided_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhaozixiang1228/mmif-ddfm/blob/HEAD/guided_diffusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"500976c0521f9f77"}},{"code_sha256_prefix":"19465fcfbb7556c2","entry":"imagefeature2textfeature","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19465fcfbb7556c2"}},{"code_sha256_prefix":"15f0af0dcf7ed33b","entry":"psf2otf","repo":"zhaozixiang1228/mmif-ddfm","repo_kind":"official","path":"guided_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhaozixiang1228/mmif-ddfm/blob/HEAD/guided_diffusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15f0af0dcf7ed33b"}},{"code_sha256_prefix":"efbaf56152135934","entry":"text_preprocess","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efbaf56152135934"}},{"code_sha256_prefix":"48e78dfab1e4912c","entry":"BaseFeatureExtraction","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48e78dfab1e4912c"}},{"code_sha256_prefix":"41504cc32fc4911f","entry":"GaussianDiffusion","repo":"zhaozixiang1228/mmif-ddfm","repo_kind":"official","path":"guided_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhaozixiang1228/mmif-ddfm/blob/HEAD/guided_diffusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41504cc32fc4911f"}},{"code_sha256_prefix":"610a6372a350c17e","entry":"Net","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"610a6372a350c17e"}},{"code_sha256_prefix":"1fc84abe149bcb05","entry":"Restormer_CNN_block","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fc84abe149bcb05"}},{"code_sha256_prefix":"1d47b40725571522","entry":"Restormer_Encoder","repo":"zhaozixiang1228/mmif-cddfuse","repo_kind":"listed","path":"net.py","file_url":"https://github.com/zhaozixiang1228/mmif-cddfuse/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d47b40725571522"}},{"code_sha256_prefix":"76102fd730986de1","entry":"Ufuser","repo":"zhaozixiang1228/mmif-emma","repo_kind":"listed","path":"nets/Ufuser.py","file_url":"https://github.com/zhaozixiang1228/mmif-emma/blob/HEAD/nets/Ufuser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76102fd730986de1"}},{"code_sha256_prefix":"e92451e6fb900818","entry":"get_mean_processor","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e92451e6fb900818"}},{"code_sha256_prefix":"0b97acea1e85e92b","entry":"get_var_processor","repo":"zhaozixiang1228/mmif-ddfm","repo_kind":"official","path":"guided_diffusion/gaussian_diffusion.py","file_url":"https://github.com/zhaozixiang1228/mmif-ddfm/blob/HEAD/guided_diffusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b97acea1e85e92b"}},{"code_sha256_prefix":"ea4e2e2f5875a2cc","entry":"restormer_cablock","repo":"zhaozixiang1228/if-film","repo_kind":"listed","path":"net/Film.py","file_url":"https://github.com/zhaozixiang1228/if-film/blob/HEAD/net/Film.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea4e2e2f5875a2cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}