{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-multimodal-fusion-with","title":"Contrastive Multimodal Fusion with TupleInfoNCE","arxiv_id":"2107.02575","date":"2021-07-06","proceeding":"ICCV 2021 10","authors":["Yunze Liu","Qingnan Fan","Shanghang Zhang","Hao Dong","Thomas Funkhouser","Li Yi"],"abstract":"This paper proposes a method for representation learning of multimodal data using contrastive losses. A traditional approach is to contrast different modalities to learn the information shared between them. However, that approach could fail to learn the complementary synergies between modalities that might be useful for downstream tasks. Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences. However, that approach could consider only the stronger modalities while ignoring the weaker ones. To address these issues, we propose a novel contrastive learning objective, TupleInfoNCE. It contrasts tuples based not only on positive and negative correspondences but also by composing new negative tuples using modalities describing different scenes. Training with these additional negatives encourages the learning model to examine the correspondences among modalities in the same tuple, ensuring that weak modalities are not ignored. We provide a theoretical justification based on mutual information for why this approach works, and we propose a sample optimization algorithm to generate positive and negative samples to maximize training efficacy. We find that TupleInfoNCE significantly outperforms the previous state of the arts on three different downstream tasks.","url_abs":"https://arxiv.org/abs/2107.02575v1","url_pdf":"https://arxiv.org/pdf/2107.02575v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-multimodal-fusion-with","repo_url":"https://github.com/hoi4d/TupleInfoNCE","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"TupleInfoNCE","rank_in_archive_order":82,"of":121,"metrics":{"Mean IoU":"48.1%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2107.02575","atlas_url":"https://app.syntology.ai/?focus=2107.02575","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.02575"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hoi4d/TupleInfoNCE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dfa3a56c484da6ab","entry":"get_best_checkpoint","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfa3a56c484da6ab"}},{"code_sha256_prefix":"5f64bc843defdf1d","entry":"get_optimizer","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f64bc843defdf1d"}},{"code_sha256_prefix":"353ebc0e90de147f","entry":"get_preprocessor","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"src/preprocessing.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/src/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"353ebc0e90de147f"}},{"code_sha256_prefix":"ecf4aca51d0a4cf5","entry":"load_ckpt","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecf4aca51d0a4cf5"}},{"code_sha256_prefix":"dc73f8421e04655c","entry":"train_one_epoch","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"search_para.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/search_para.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc73f8421e04655c"}},{"code_sha256_prefix":"ab8055f9e891a387","entry":"validate","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"search_para.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/search_para.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab8055f9e891a387"}},{"code_sha256_prefix":"0e04eede81ffcfae","entry":"validate_sampling","repo":"hoi4d/TupleInfoNCE","repo_kind":"official","path":"search_para.py","file_url":"https://github.com/hoi4d/TupleInfoNCE/blob/HEAD/search_para.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e04eede81ffcfae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}