{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tip-tabular-image-pre-training-for-multimodal","title":"TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data","arxiv_id":"2407.07582","date":"2024-07-10","proceeding":null,"authors":["Siyi Du","Shaoming Zheng","Yinsong Wang","Wenjia Bai","Declan P. O'Regan","Chen Qin"],"abstract":"Images and structured tables are essential parts of real-world databases. Though tabular-image representation learning is promising to create new insights, it remains a challenging task, as tabular data is typically heterogeneous and incomplete, presenting significant modality disparities with images. Earlier works have mainly focused on simple modality fusion strategies in complete data scenarios, without considering the missing data issue, and thus are limited in practice. In this paper, we propose TIP, a novel tabular-image pre-training framework for learning multimodal representations robust to incomplete tabular data. Specifically, TIP investigates a novel self-supervised learning (SSL) strategy, including a masked tabular reconstruction task for tackling data missingness, and image-tabular matching and contrastive learning objectives to capture multimodal information. Moreover, TIP proposes a versatile tabular encoder tailored for incomplete, heterogeneous tabular data and a multimodal interaction module for inter-modality representation learning. Experiments are performed on downstream multimodal classification tasks using both natural and medical image datasets. The results show that TIP outperforms state-of-the-art supervised/SSL image/multimodal algorithms in both complete and incomplete data scenarios. Our code is available at https://github.com/siyi-wind/TIP.","url_abs":"https://arxiv.org/abs/2407.07582v1","url_pdf":"https://arxiv.org/pdf/2407.07582v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tip-tabular-image-pre-training-for-multimodal","repo_url":"https://github.com/siyi-wind/tip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"multimodal-interaction","task_name":"multimodal interaction"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.07582","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07582"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/siyi-wind/TIP","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"siyi-wind/TIP","repo_kind":"official","path":"models/Tip_utils/ResNet.py","file_url":"https://github.com/siyi-wind/TIP/blob/HEAD/models/Tip_utils/ResNet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"099701328b673956","entry":"load_pretrain","repo":"siyi-wind/TIP","repo_kind":"official","path":"models/Tip_utils/ResNet.py","file_url":"https://github.com/siyi-wind/TIP/blob/HEAD/models/Tip_utils/ResNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"099701328b673956"}},{"code_sha256_prefix":"1444cafc7b022c45","entry":"create_vit","repo":"siyi-wind/TIP","repo_kind":"official","path":"models/Tip_utils/VisionTransformer_imagenet.py","file_url":"https://github.com/siyi-wind/TIP/blob/HEAD/models/Tip_utils/VisionTransformer_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1444cafc7b022c45"}},{"code_sha256_prefix":"16ac3b5df0d00348","entry":"resnet18","repo":"siyi-wind/TIP","repo_kind":"official","path":"models/Tip_utils/ResNet.py","file_url":"https://github.com/siyi-wind/TIP/blob/HEAD/models/Tip_utils/ResNet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16ac3b5df0d00348"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}