{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/text-difuse-an-interactive-multi-modal-image","title":"Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model","arxiv_id":"2410.23905","date":"2024-10-31","proceeding":null,"authors":["Hao Zhang","Lei Cao","Jiayi Ma"],"abstract":"Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, \\textit{etc}. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.","url_abs":"https://arxiv.org/abs/2410.23905v1","url_pdf":"https://arxiv.org/pdf/2410.23905v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"text-difuse-an-interactive-multi-modal-image","repo_url":"https://github.com/leiii-cao/text-difuse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.23905","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.23905"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Leiii-Cao/Text-DiFuse","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leiii-cao/text-difuse","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_honours":1,"ran_draft_wrong":2,"ran_fixture":1,"unverified":17},"by_repo_kind":{"official":{"samples":25,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2aaae79c9ede0635","entry":"CBAM","repo":"leiii-cao/text-difuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/leiii-cao/text-difuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aaae79c9ede0635"}},{"code_sha256_prefix":"4b4b3d10f2b6b2ea","entry":"ChannelAttentionModule","repo":"leiii-cao/text-difuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/leiii-cao/text-difuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4b3d10f2b6b2ea"}},{"code_sha256_prefix":"7d354c931a8b73fb","entry":"Get_Fusion_Control_Model","repo":"leiii-cao/text-difuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/leiii-cao/text-difuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d354c931a8b73fb"}},{"code_sha256_prefix":"3e2d5f9fbded1106","entry":"SpatialAttentionModule","repo":"leiii-cao/text-difuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/leiii-cao/text-difuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e2d5f9fbded1106"}},{"code_sha256_prefix":"a086d6286a40b889","entry":"get_named_beta_schedule","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/gaussian_diffusion.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/gaussian_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a086d6286a40b889"}},{"code_sha256_prefix":"2d450c485d6f0636","entry":"group_norm","repo":"leiii-cao/text-difuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/leiii-cao/text-difuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d450c485d6f0636"}},{"code_sha256_prefix":"4707efa98ac6b989","entry":"normal_kl","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/losses.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/losses.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4707efa98ac6b989"}},{"code_sha256_prefix":"129b804760b3115f","entry":"zero_module","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/nn_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/nn_util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"129b804760b3115f"}},{"code_sha256_prefix":"bd2216dfcf7872aa","entry":"Get_feature_all","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/gaussian_diffusion.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd2216dfcf7872aa"}},{"code_sha256_prefix":"4e5dd44f6551d1d8","entry":"Spit_feature_all","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/gaussian_diffusion.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/gaussian_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e5dd44f6551d1d8"}},{"code_sha256_prefix":"f15aa225ff9ad213","entry":"approx_standard_normal_cdf","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/losses.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f15aa225ff9ad213"}},{"code_sha256_prefix":"ecd0fc28815b65ae","entry":"avg_pool_nd","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/nn_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/nn_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecd0fc28815b65ae"}},{"code_sha256_prefix":"fe4eb545bbb728e0","entry":"conv_nd","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/nn_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/nn_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe4eb545bbb728e0"}},{"code_sha256_prefix":"d9013b673760ac7c","entry":"create_model_and_diffusion","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/script_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/script_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9013b673760ac7c"}},{"code_sha256_prefix":"34431d326a1db3be","entry":"create_named_schedule_sampler","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/resample.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/resample.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34431d326a1db3be"}},{"code_sha256_prefix":"667496bffcb3922d","entry":"create_window","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"667496bffcb3922d"}},{"code_sha256_prefix":"6e228e6f353adfc1","entry":"detect_objects","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"modulated/util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/modulated/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e228e6f353adfc1"}},{"code_sha256_prefix":"630ea2365cd982bf","entry":"diffusion_stage1_create_model","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/script_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/script_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"630ea2365cd982bf"}},{"code_sha256_prefix":"59ec9ffb7aaf338a","entry":"diffusion_stage2_create_model","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/script_util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/script_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59ec9ffb7aaf338a"}},{"code_sha256_prefix":"9665e15b8cd77a32","entry":"discretized_gaussian_log_likelihood","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/losses.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9665e15b8cd77a32"}},{"code_sha256_prefix":"c813c3992f7b56da","entry":"gaussian","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c813c3992f7b56da"}},{"code_sha256_prefix":"eb656fe08a6a70c4","entry":"is_image_file","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb656fe08a6a70c4"}},{"code_sha256_prefix":"b930371b74e318b8","entry":"load_owlvit","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"modulated/util.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/modulated/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b930371b74e318b8"}},{"code_sha256_prefix":"48416fe6fb1e3520","entry":"space_timesteps","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/respace.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/respace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48416fe6fb1e3520"}},{"code_sha256_prefix":"6ddd349a862de98b","entry":"time_embedding","repo":"Leiii-Cao/Text-DiFuse","repo_kind":"official","path":"diffusion_fusion/unet.py","file_url":"https://github.com/Leiii-Cao/Text-DiFuse/blob/HEAD/diffusion_fusion/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ddd349a862de98b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}