{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quantifying-and-enhancing-multi-modal","title":"Quantifying and Enhancing Multi-modal Robustness with Modality Preference","arxiv_id":"2402.06244","date":"2024-02-09","proceeding":null,"authors":["Zequn Yang","Yake Wei","Ce Liang","Di Hu"],"abstract":"Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.","url_abs":"https://arxiv.org/abs/2402.06244v2","url_pdf":"https://arxiv.org/pdf/2402.06244v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quantifying-and-enhancing-multi-modal","repo_url":"https://github.com/gewu-lab/certifiable-robust-multi-modal-training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.06244","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.06244"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gewu-lab/certifiable-robust-multi-modal-training","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"f445cc20cc72c61a","entry":"OWNNorm","repo":"gewu-lab/certifiable-robust-multi-modal-training","repo_kind":"official","path":"models/fusion_model.py","file_url":"https://github.com/gewu-lab/certifiable-robust-multi-modal-training/blob/HEAD/models/fusion_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f445cc20cc72c61a"}},{"code_sha256_prefix":"c96b174740a92073","entry":"OrthFusion","repo":"gewu-lab/certifiable-robust-multi-modal-training","repo_kind":"official","path":"models/fusion_model.py","file_url":"https://github.com/gewu-lab/certifiable-robust-multi-modal-training/blob/HEAD/models/fusion_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c96b174740a92073"}},{"code_sha256_prefix":"31ed62bbc034a89f","entry":"calculate_accuracy","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31ed62bbc034a89f"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"models/backbone.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/models/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"models/backbone.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/models/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"a11eecab8e7e04af","entry":"get_lr","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a11eecab8e7e04af"}},{"code_sha256_prefix":"5dfe866366f3d5eb","entry":"inv_norm_tensor","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"datasets/KS_dataset.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/datasets/KS_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5dfe866366f3d5eb"}},{"code_sha256_prefix":"158c46d01644f4c4","entry":"orthogonal_linear","repo":"gewu-lab/certifiable-robust-multi-modal-training","repo_kind":"official","path":"models/fusion_model.py","file_url":"https://github.com/gewu-lab/certifiable-robust-multi-modal-training/blob/HEAD/models/fusion_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"158c46d01644f4c4"}},{"code_sha256_prefix":"0ab7cbcb8ecef516","entry":"partialclass","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ab7cbcb8ecef516"}},{"code_sha256_prefix":"7cfced1350c1f62a","entry":"cal_for_frames","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"pre-processing/extract_rgb_of.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/pre-processing/extract_rgb_of.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cfced1350c1f62a"}},{"code_sha256_prefix":"1f7c7759ae51be72","entry":"compute_TVL1","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"pre-processing/extract_rgb_of.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/pre-processing/extract_rgb_of.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f7c7759ae51be72"}},{"code_sha256_prefix":"e7f8e892b97e7abf","entry":"cross_modality_pretrain","repo":"GeWu-Lab/Certifiable-Robust-Multi-modal-Training","repo_kind":"official","path":"models/backbone.py","file_url":"https://github.com/GeWu-Lab/Certifiable-Robust-Multi-modal-Training/blob/HEAD/models/backbone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7f8e892b97e7abf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}