Multimodal Deep Learning

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

22 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 97 papers with code (213 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
  28 Mar 2023 7 repositories listed
  We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model.
• LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
  3 Oct 2023 6 repositories listed Syntology ran 7 of 14 samples · 7 unverified
  We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M.
• ImageBind: One Embedding Space To Bind Them All
  9 May 2023 3 repositories listed Syntology ran 24 of 34 samples · 10 unverified · 32 pointer-only (licence)
  We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together.
• aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception
  17 Nov 2022 3 repositories listed Syntology ran 0 of 5 samples · 5 unverified
  The dataset consists of 176 scenes with synchronized and calibrated LiDAR, camera, and radar sensors covering a 360-degree field of view.
• Supervised Video Summarization via Multiple Feature Sets with Parallel Attention
  23 Apr 2021 3 repositories listed
  The proposed architecture utilizes an attention mechanism before fusing motion features and features representing the (static) visual content, i.
• Are These Birds Similar: Learning Branched Networks for Fine-grained Representations
  16 Jan 2020 3 repositories listed
  In recent years, natural language descriptions are used to obtain information on discriminative parts of the object.
• Multimodal deep networks for text and image-based document classification
  15 Jul 2019 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures.
• ShapeWorld - A new test methodology for multimodal language understanding
  14 Apr 2017 3 repositories listed
  We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities.
• ViTextVQA: A Large-Scale Visual Question Answering Dataset for Evaluating Vietnamese Text Comprehension in Images
  16 Apr 2024 2 repositories listed
  Therefore, we introduce the first large-scale dataset in Vietnamese specializing in the ability to understand scene text, we call it ViTextVQA (\textbf{Vi}etnamese \textbf{Text}-based \textbf{V}isual \textbf{Q}uestion…
• HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
  15 Nov 2023 2 repositories listed Syntology ran 6 of 11 samples · 5 unverified
  Technological advances in medical data collection, such as high-throughput genomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling,…
• Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning
  11 Nov 2023 2 repositories listed
  The versatility of multimodal deep learning holds tremendous promise for advancing scientific research and practical applications.
• HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk Factors
  2 Oct 2023 2 repositories listed
  Our MMDL system uses RETFound, a foundation model pre-trained on 1.
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
  28 Jul 2021 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data.
• A multimodal deep learning framework for scalable content based visual media retrieval
  18 May 2021 2 repositories listed
  We propose a novel, efficient, modular and scalable framework for content based visual media retrieval systems by leveraging the power of Deep Learning which is flexible to work both for images and videos conjointly and…
• Multimodal Deep Learning for Robust RGB-D Object Recognition
  24 Jul 2015 2 repositories listed
  Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications.
• Gaze-Guided Learning: Avoiding Shortcut Bias in Visual Classification
  8 Apr 2025 1 repository listed
  Motivated by this, we introduce Gaze-CIFAR-10, a human gaze time-series dataset, along with a dual-sequence gaze encoder that models the precise sequential localization of human attention on distinct local attributes.
• Multimodal Deep Learning for Subtype Classification in Breast Cancer Using Histopathological Images and Gene Expression Data
  4 Mar 2025 1 repository listed
  Molecular subtyping of breast cancer is crucial for personalized treatment and prognosis.
• A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions
  9 Feb 2025 1 repository listed
  PDE foundation models utilize neural networks to train approximations to multiple differential equations simultaneously and are thus a general purpose solver that can be adapted to downstream tasks.
• Multimodal Marvels of Deep Learning in Medical Diagnosis: A Comprehensive Review of COVID-19 Detection
  16 Jan 2025 1 repository listed
  Our analysis revealed that the MobileNet model achieved the highest accuracy of 99.
• CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval
  17 Dec 2024 1 repository listed
  This unified lightweight model bridges the gap between various modalities and languages, enhancing its effectiveness in handling and retrieving multilingual and multimodal data.
• Frozen Large-scale Pretrained Vision-Language Models are the Effective Foundational Backbone for Multimodal Breast Cancer Prediction
  27 Nov 2024 1 repository listed
  This study introduces a multimodal deep-learning model leveraging mammogram datasets to evaluate breast cancer prediction.
• CardioLab: Laboratory Values Estimation and Monitoring from Electrocardiogram Signals -- A Multimodal Deep Learning Approach
  22 Nov 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  Conclusions: This study provides first evidence for the feasibility of using ECG data alongside clinical routine data for the real-time estimation and monitoring of laboratory value abnormalities, which could provide a…
• Uncovering the Genetic Basis of Glioblastoma Heterogeneity through Multimodal Analysis of Whole Slide Images and RNA Sequencing Data
  23 Oct 2024 1 repository listed
  Glioblastoma is a highly aggressive form of brain cancer characterized by rapid progression and poor prognosis.
• PHemoNet: A Multimodal Network for Physiological Signals
  13 Sep 2024 1 repository listed
  In this paper, we introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals.
• Dual-Level Cross-Modal Contrastive Clustering
  6 Sep 2024 1 repository listed
  However, there is a gap between visual representation learning and textual semantic learning, and how to properly utilize the representation of two different modalities for clustering is still a big challenge.
• MVX-ViT: Multimodal Collaborative Perception for 6G V2X Network Management Decisions Using Vision Transformer.
  30 Aug 2024 1 repository listed
  Advancements in sixth-generation (6G) networks, coupled with the evolution of multimodal sensing in vehicle-to-everything (V2X) networks, have opened avenues for transformative research into multimodal-based artificial…
• Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading
  16 Aug 2024 1 repository listed
  Remarkably, our FoF achieves superior performance using only histopathology slides compared to existing multimodal methods.
• Modeling of spatially embedded networks via regional spatial graph convolutional networks
  20 Jun 2024 1 repository listed
  This study introduces a novel geometric-based multimodal deep learning model for spatially embedded network representation learning, namely the regional spatial graph convolutional network (RSGCN).
• LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data
  14 Jun 2024 1 repository listed
  It extends the well-known CIFAR 10/100 dataset with audio samples extracted from three audio corpora, and text data generated using the Gemma-7B Large Language Model (LLM).
• Automatic Fused Multimodal Deep Learning for Plant Identification
  3 Jun 2024 1 repository listed
  In this paper, we introduce a pioneering multimodal DL-based approach for plant classification with automatic modality fusion.

Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.