{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gramian-multimodal-representation-learning","title":"Gramian Multimodal Representation Learning and Alignment","arxiv_id":"2412.11959","date":"2024-12-16","proceeding":null,"authors":["Giordano Cicchetti","Eleonora Grassucci","Luigi Sigillo","Danilo Comminiello"],"abstract":"Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modality and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code, and the pretrained models are available at https://ispamm.github.io/GRAM/.","url_abs":"https://arxiv.org/abs/2412.11959v1","url_pdf":"https://arxiv.org/pdf/2412.11959v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gramian-multimodal-representation-learning","repo_url":"https://github.com/ispamm/GRAM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gramian-multimodal-representation-learning","repo_url":"https://github.com/luigisigillo/gwit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"video-classification","task_name":"Video Classification"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"zero-shot-audio-retrieval","task_name":"Zero-Shot Audio Retrieval"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"GRAM","rank_in_archive_order":4,"of":31,"metrics":{"text-to-video R@1":"69.9","text-to-video R@10":"96.1","video-to-text R@1":"66.9","video-to-text R@10":"95.4"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"GRAM","rank_in_archive_order":6,"of":40,"metrics":{"text-to-video R@1":"67.3","text-to-video R@10":"90.1","video-to-text R@1":"63.5","video-to-text R@10":"91.6"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"GRAM","rank_in_archive_order":1,"of":40,"metrics":{"text-to-video R@1":"64","text-to-video R@10":"89.3","video-to-text R@1":"64.8","video-to-text R@10":"91.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-vatex","task":"Video Retrieval","dataset":"VATEX","model":"GRAM","rank_in_archive_order":1,"of":13,"metrics":{"text-to-video R@1":"87.7","text-to-video R@10":"100","video-to-text R@1":"84.6","video-to-text R@10":"100"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"GRAM","rank_in_archive_order":3,"of":12,"metrics":{"text-to-video R@1":"59.0","text-to-video R@10":"91.2","video-to-text R@1":"50.9","video-to-text R@10":"85.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"GRAM","rank_in_archive_order":4,"of":26,"metrics":{"text-to-video R@1":"54.2","text-to-video R@10":"80.7","video-to-text R@1":"52.3","video-to-text R@10":"80.3"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"GRAM","rank_in_archive_order":2,"of":41,"metrics":{"text-to-video R@1":"54.8","text-to-video R@10":"83.9","video-to-text R@1":"52.9","video-to-text R@10":"82.9"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-vatex","task":"Zero-Shot Video Retrieval","dataset":"VATEX","model":"GRAM","rank_in_archive_order":1,"of":5,"metrics":{"text-to-video R@1":"83.9","text-to-video R@10":"99.5","video-to-text R@1":"82.7","video-to-text R@10":"99"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.11959","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.11959"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luigisigillo/gwit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ispamm/GRAM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_honours":1,"unverified":10},"by_repo_kind":{"official":{"samples":12,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"ispamm/GRAM","repo_kind":"official","path":"model/general_module.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/model/general_module.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"ispamm/GRAM","repo_kind":"official","path":"model/general_module.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/model/general_module.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"eee800bfe0a47a9f","entry":"all_gather_list","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/distributed.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee800bfe0a47a9f"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/distributed.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"cc4ffe92873b4ee6","entry":"build_optimizer","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/build_optimizer.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/build_optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc4ffe92873b4ee6"}},{"code_sha256_prefix":"0122d74913461cf9","entry":"compute_max_audio_sample_num_for_position_embeddings","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/args.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/args.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0122d74913461cf9"}},{"code_sha256_prefix":"30fd4a1a3eedb19c","entry":"compute_max_vision_sample_num_for_position_embeddings","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/args.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/args.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30fd4a1a3eedb19c"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/distributed.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"77a45c1e515dbe2e","entry":"execCmd","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/offline_process_data.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/offline_process_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77a45c1e515dbe2e"}},{"code_sha256_prefix":"7d297bdebd282cba","entry":"load_from_pretrained_dir","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/build_model.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/build_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d297bdebd282cba"}},{"code_sha256_prefix":"647f671c01bde501","entry":"load_from_resume","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/build_model.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/build_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"647f671c01bde501"}},{"code_sha256_prefix":"15b65a6a6e8fd954","entry":"parse_with_config","repo":"ispamm/GRAM","repo_kind":"official","path":"utils/args.py","file_url":"https://github.com/ispamm/GRAM/blob/HEAD/utils/args.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15b65a6a6e8fd954"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}