{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-interactive-feature-learning-and-a-full","title":"Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation","arxiv_id":"2308.02097","date":"2023-08-04","proceeding":"ICCV 2023 1","authors":["JinYuan Liu","Zhu Liu","Guanyao Wu","Long Ma","Risheng Liu","Wei Zhong","Zhongxuan Luo","Xin Fan"],"abstract":"Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \\emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \\textbf{M}ulti-\\textbf{i}nteractive \\textbf{F}eature learning architecture for image fusion and \\textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \\url{https://github.com/JinyuanLiu-CV/SegMiF}.","url_abs":"https://arxiv.org/abs/2308.02097v1","url_pdf":"https://arxiv.org/pdf/2308.02097v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-interactive-feature-learning-and-a-full","repo_url":"https://github.com/jinyuanliu-cv/segmif","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"multi-interactive-feature-learning-and-a-full","repo_url":"https://github.com/runjia0124/coconet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[{"slug":"fmb-dataset","name":"FMB Dataset","full_name":"Full-time Multi-modality Benchmark Dataset"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-fmb-dataset","task":"Semantic Segmentation","dataset":"FMB Dataset","model":"SegMiF (RGB-Infrared)","rank_in_archive_order":7,"of":14,"metrics":{"mIoU":"54.80"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-fmb-dataset","task":"Semantic Segmentation","dataset":"FMB Dataset","model":"SegMiF (RGB)","rank_in_archive_order":10,"of":14,"metrics":{"mIoU":"50.50"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-mfn-dataset","task":"Thermal Image Segmentation","dataset":"MFN Dataset","model":"SegMiF","rank_in_archive_order":25,"of":55,"metrics":{"mIOU":"56.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.02097","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.02097"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/runjia0124/coconet","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinyuanliu-cv/segmif","reach":{"status":"ok"}}],"summary":{"ran":18,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1},"listed":{"samples":13,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"92167228343b84a5","entry":"YCbCr2RGB","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92167228343b84a5"}},{"code_sha256_prefix":"145edd13c3925a58","entry":"YCrCb2RGB","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"core/model_fusion.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/core/model_fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"145edd13c3925a58"}},{"code_sha256_prefix":"c567234c5e7fa848","entry":"compute_cls_loss","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"core/loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c567234c5e7fa848"}},{"code_sha256_prefix":"2fe6b97c7e7cccbb","entry":"compute_cnt_loss","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"core/loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fe6b97c7e7cccbb"}},{"code_sha256_prefix":"68ee569246bd7e3a","entry":"coords_fmap2orig","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"core/loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68ee569246bd7e3a"}},{"code_sha256_prefix":"1f6001e1c3b06556","entry":"downsample","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"lap_loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/lap_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f6001e1c3b06556"}},{"code_sha256_prefix":"71a890b56b7f51ea","entry":"fname_presuffix","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71a890b56b7f51ea"}},{"code_sha256_prefix":"4a150d48b91667f9","entry":"gauss_kernel","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"lap_loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/lap_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a150d48b91667f9"}},{"code_sha256_prefix":"0cc9285c14985e5d","entry":"load_img_name_list","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"datasets/voc_method.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/datasets/voc_method.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cc9285c14985e5d"}},{"code_sha256_prefix":"89e4412c100b0f9b","entry":"normPRED","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/save_image.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/save_image.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89e4412c100b0f9b"}},{"code_sha256_prefix":"d1a6979f946f9e74","entry":"normalize_img","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"datasets/imutils.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/datasets/imutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1a6979f946f9e74"}},{"code_sha256_prefix":"df54d7f9e9e4ab99","entry":"normimage","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/save_image.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/save_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df54d7f9e9e4ab99"}},{"code_sha256_prefix":"1d18356aa2fee4e0","entry":"normimage_test","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/save_image.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/save_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d18356aa2fee4e0"}},{"code_sha256_prefix":"b79bae2be65a7ef0","entry":"prepare_data_path","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"TaskFusion_dataset2.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/TaskFusion_dataset2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b79bae2be65a7ef0"}},{"code_sha256_prefix":"a920d698dd17a625","entry":"random_scaling","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"datasets/imutils.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/datasets/imutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a920d698dd17a625"}},{"code_sha256_prefix":"49fc95c7acdb2ea4","entry":"random_scaling2","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"datasets/imutils.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/datasets/imutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49fc95c7acdb2ea4"}},{"code_sha256_prefix":"47482689d9a75517","entry":"split_filename","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47482689d9a75517"}},{"code_sha256_prefix":"267b7d002ecac955","entry":"upsample","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"lap_loss.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/lap_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"267b7d002ecac955"}},{"code_sha256_prefix":"b683a315ccc25715","entry":"weights_to_cpu","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/checkpoint.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b683a315ccc25715"}},{"code_sha256_prefix":"6b1c47ea9127b33e","entry":"RGB2YCrCb","repo":"jinyuanliu-cv/segmif","repo_kind":"official","path":"core/model_fusion.py","file_url":"https://github.com/jinyuanliu-cv/segmif/blob/HEAD/core/model_fusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b1c47ea9127b33e"}},{"code_sha256_prefix":"bbfe8d248ce2ed69","entry":"load_model","repo":"runjia0124/coconet","repo_kind":"listed","path":"models/P_loss.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/models/P_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbfe8d248ce2ed69"}},{"code_sha256_prefix":"0511b91fcab989b2","entry":"load_part","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/checkpoint.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0511b91fcab989b2"}},{"code_sha256_prefix":"76e6bd620ca82ccc","entry":"measure_info","repo":"runjia0124/coconet","repo_kind":"listed","path":"models/train_tasks.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/models/train_tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76e6bd620ca82ccc"}},{"code_sha256_prefix":"4f4b727b0fc6cd95","entry":"measure_module1","repo":"runjia0124/coconet","repo_kind":"listed","path":"models/train_tasks.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/models/train_tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f4b727b0fc6cd95"}},{"code_sha256_prefix":"a3db2d0cc62c57ce","entry":"resume","repo":"runjia0124/coconet","repo_kind":"listed","path":"utils/checkpoint.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/utils/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3db2d0cc62c57ce"}},{"code_sha256_prefix":"7fbf8b3d61a9af4b","entry":"softmax","repo":"runjia0124/coconet","repo_kind":"listed","path":"models/train_tasks.py","file_url":"https://github.com/runjia0124/coconet/blob/HEAD/models/train_tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fbf8b3d61a9af4b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}