{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/paif-perception-aware-infrared-visible-image","title":"PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation","arxiv_id":"2308.03979","date":"2023-08-08","proceeding":null,"authors":["Zhu Liu","JinYuan Liu","Benzhuang Zhang","Long Ma","Xin Fan","Risheng Liu"],"abstract":"Infrared and visible image fusion is a powerful technique that combines complementary information from different modalities for downstream semantic perception tasks. Existing learning-based methods show remarkable performance, but are suffering from the inherent vulnerability of adversarial attacks, causing a significant decrease in accuracy. In this work, a perception-aware fusion framework is proposed to promote segmentation robustness in adversarial scenes. We first conduct systematic analyses about the components of image fusion, investigating the correlation with segmentation robustness under adversarial perturbations. Based on these analyses, we propose a harmonized architecture search with a decomposition-based structure to balance standard accuracy and robustness. We also propose an adaptive learning strategy to improve the parameter robustness of image fusion, which can learn effective feature extraction under diverse adversarial perturbations. Thus, the goals of image fusion (\\textit{i.e.,} extracting complementary features from source modalities and defending attack) can be realized from the perspectives of architectural and learning strategies. Extensive experimental results demonstrate that our scheme substantially enhances the robustness, with gains of 15.3% mIOU of segmentation in the adversarial scene, compared with advanced competitors. The source codes are available at https://github.com/LiuZhu-CV/PAIF.","url_abs":"https://arxiv.org/abs/2308.03979v1","url_pdf":"https://arxiv.org/pdf/2308.03979v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"paif-perception-aware-infrared-visible-image","repo_url":"https://github.com/liuzhu-cv/paif","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"paif-perception-aware-infrared-visible-image","repo_url":"https://github.com/LiuZhu-CV/BDLFusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"paif-perception-aware-infrared-visible-image","repo_url":"https://github.com/liuzhu-cv/crmef","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"infrared-and-visible-image-fusion","task_name":"Infrared And Visible Image Fusion"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/thermal-image-segmentation-on-mfn-dataset","task":"Thermal Image Segmentation","dataset":"MFN Dataset","model":"PAIF","rank_in_archive_order":22,"of":55,"metrics":{"mIOU":"56.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.03979","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.03979"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuzhu-cv/crmef","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuzhu-cv/paif","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LiuZhu-CV/BDLFusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11,"ran_draft_wrong":2,"unverified":14},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1},"listed":{"samples":25,"ran":13,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"0415a04e36607241","entry":"CropSample","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"pietorch/N_modules.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/pietorch/N_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0415a04e36607241"}},{"code_sha256_prefix":"79504595f406b0de","entry":"CropSample2","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"pietorch/N_modules.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/pietorch/N_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79504595f406b0de"}},{"code_sha256_prefix":"747145371dbd7ffb","entry":"CropSample3","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"pietorch/N_modules.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/pietorch/N_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"747145371dbd7ffb"}},{"code_sha256_prefix":"145edd13c3925a58","entry":"YCrCb2RGB","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/fcos_fusion.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/fcos_fusion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"145edd13c3925a58"}},{"code_sha256_prefix":"0c6f4d7bb0001241","entry":"cal_gradient_penalty","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"AHDRNAS.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/AHDRNAS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c6f4d7bb0001241"}},{"code_sha256_prefix":"c567234c5e7fa848","entry":"compute_cls_loss","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/loss.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c567234c5e7fa848"}},{"code_sha256_prefix":"2fe6b97c7e7cccbb","entry":"compute_cnt_loss","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/loss.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fe6b97c7e7cccbb"}},{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/backbone/resnet.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/backbone/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"583f9780bdd00a45","entry":"conv3x3","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"pietorch/se_nets.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/pietorch/se_nets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"583f9780bdd00a45"}},{"code_sha256_prefix":"d3a234f32905b183","entry":"conv_layer","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"AHDRNAS.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/AHDRNAS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3a234f32905b183"}},{"code_sha256_prefix":"68ee569246bd7e3a","entry":"coords_fmap2orig","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/loss.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68ee569246bd7e3a"}},{"code_sha256_prefix":"6a4bda1402e49df9","entry":"get_patch_from_file","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"dataset/dataset_patches.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/dataset/dataset_patches.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a4bda1402e49df9"}},{"code_sha256_prefix":"1a58de2bd0d27f53","entry":"get_tv","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"PerceptualLoss.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/PerceptualLoss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a58de2bd0d27f53"}},{"code_sha256_prefix":"f724ee102895d8e3","entry":"RGB2YCrCb","repo":"liuzhu-cv/paif","repo_kind":"official","path":"test_original.py","file_url":"https://github.com/liuzhu-cv/paif/blob/HEAD/test_original.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f724ee102895d8e3"}},{"code_sha256_prefix":"6b1c47ea9127b33e","entry":"RGB2YCrCb","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/fcos_fusion.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/fcos_fusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b1c47ea9127b33e"}},{"code_sha256_prefix":"96dcdaf98db413c7","entry":"YCrCb2RGB","repo":"liuzhu-cv/paif","repo_kind":"official","path":"test_original.py","file_url":"https://github.com/liuzhu-cv/paif/blob/HEAD/test_original.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96dcdaf98db413c7"}},{"code_sha256_prefix":"8995c32130a77e1b","entry":"colorJitter","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"dataset/augment.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/dataset/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8995c32130a77e1b"}},{"code_sha256_prefix":"0db1cd12ba9b89e6","entry":"data_augmentation","repo":"liuzhu-cv/crmef","repo_kind":"listed","path":"dataset/dataset_patches.py","file_url":"https://github.com/liuzhu-cv/crmef/blob/HEAD/dataset/dataset_patches.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0db1cd12ba9b89e6"}},{"code_sha256_prefix":"f1eaef8506fafbc4","entry":"eval_ap_2d","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"eval_detection.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/eval_detection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1eaef8506fafbc4"}},{"code_sha256_prefix":"f282dfb509f5c783","entry":"flip","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"dataset/VOC_dataset.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/dataset/VOC_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f282dfb509f5c783"}},{"code_sha256_prefix":"77ccaaed6fe6ba6c","entry":"flip","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"dataset/VOC_dataset_Fusion.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/dataset/VOC_dataset_Fusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77ccaaed6fe6ba6c"}},{"code_sha256_prefix":"c756c28cd59b62a3","entry":"iou_2d","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"eval_detection.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/eval_detection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c756c28cd59b62a3"}},{"code_sha256_prefix":"7a9983c45529c7ed","entry":"random_rotation","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"dataset/augment.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/dataset/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a9983c45529c7ed"}},{"code_sha256_prefix":"6356ce97cd733a2e","entry":"random_rotation2","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"dataset/augment.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/dataset/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6356ce97cd733a2e"}},{"code_sha256_prefix":"004e5b2448286424","entry":"resnet18","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/backbone/resnet.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/backbone/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"004e5b2448286424"}},{"code_sha256_prefix":"ab6897aaaf1e833e","entry":"resnet34","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"model/backbone/resnet.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/model/backbone/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab6897aaaf1e833e"}},{"code_sha256_prefix":"4f5c46171ffa62aa","entry":"sort_by_score","repo":"LiuZhu-CV/BDLFusion","repo_kind":"listed","path":"eval_detection.py","file_url":"https://github.com/LiuZhu-CV/BDLFusion/blob/HEAD/eval_detection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f5c46171ffa62aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}