{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-modality-generalization-a-benchmark","title":"Towards Modality Generalization: A Benchmark and Prospective Analysis","arxiv_id":"2412.18277","date":"2024-12-24","proceeding":null,"authors":["Xiaohao Liu","Xiaobo Xia","Zhuo Huang","Tat-Seng Chua"],"abstract":"Multi-modal learning has achieved remarkable success by integrating information from various modalities, achieving superior performance in tasks like recognition and retrieval compared to uni-modal approaches. However, real-world scenarios often present novel modalities that are unseen during training due to resource and privacy constraints, a challenge current methods struggle to address. This paper introduces Modality Generalization (MG), which focuses on enabling models to generalize to unseen modalities. We define two cases: weak MG, where both seen and unseen modalities can be mapped into a joint embedding space via existing perceptors, and strong MG, where no such mappings exist. To facilitate progress, we propose a comprehensive benchmark featuring multi-modal algorithms and adapt existing methods that focus on generalization. Extensive experiments highlight the complexity of MG, exposing the limitations of existing methods and identifying key directions for future research. Our work provides a foundation for advancing robust and adaptable multi-modal models, enabling them to handle unseen modalities in realistic scenarios.","url_abs":"https://arxiv.org/abs/2412.18277v1","url_pdf":"https://arxiv.org/pdf/2412.18277v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-modality-generalization-a-benchmark","repo_url":"https://github.com/Xiaohao-Liu/ModalBed","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.18277","atlas_url":"https://app.syntology.ai/?focus=2412.18277","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.18277"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Xiaohao-Liu/ModalBed","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"d6f029f55279e5a6","entry":"format_latex_name","repo":"Xiaohao-Liu/ModalBed","repo_kind":"official","path":"modalbed/scripts/collect_results.py","file_url":"https://github.com/Xiaohao-Liu/ModalBed/blob/HEAD/modalbed/scripts/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6f029f55279e5a6"}},{"code_sha256_prefix":"49d12b3b06e55660","entry":"format_perceptor_name","repo":"Xiaohao-Liu/ModalBed","repo_kind":"official","path":"modalbed/scripts/collect_results.py","file_url":"https://github.com/Xiaohao-Liu/ModalBed/blob/HEAD/modalbed/scripts/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49d12b3b06e55660"}},{"code_sha256_prefix":"dfa58e67e2bd207e","entry":"remove_key","repo":"Xiaohao-Liu/ModalBed","repo_kind":"official","path":"modalbed/scripts/collect_results.py","file_url":"https://github.com/Xiaohao-Liu/ModalBed/blob/HEAD/modalbed/scripts/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfa58e67e2bd207e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}