Browse State-of-the-Art › Multi-modal Classification

Multi-modal Classification

12 papers with code · 2 benchmarks · 4 datasets archive 2025-07-28

Miscellaneous

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

DatasetBest model (first row in archive order)PaperCodeSyntologyCompare
VGG-Sound (4 rows) MMT Multiscale Multimodal Transformer for Multimodal Action Recognition — — Compare
AudioSet (2 rows) CAV-MAE Contrastive Audio-Visual Masked Autoencoder code Syntology official: harvested, nothing ran · 0 ran · 1 unverified Compare

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

4 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Papers archive 2025-07-28

archive papers tagged: 31 · with a code link: 12 · where Syntology ran a sample: 3 (2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (3 of 31 tagged: 2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 31 of 31, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.

  • Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal…
  • 23 Dec 2024 1 repository listed
    Multimodal AI models are increasingly used in fields like healthcare, finance, and autonomous driving, where information is drawn from multiple sources or modalities such as images, texts, audios, videos.
  • 13 Nov 2024 1 repository listed
    The prevalence of multi-modal content on social media complicates automated moderation strategies.
  • In a joint vision-language space, a text feature (e. g., from "a photo of a dog") could effectively represent its relevant image features (e. g., from dog photos).
  • 4 Mar 2023 1 repository listed
    In the fashion domain, there exists a variety of vision-and-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning.
  • 2 Oct 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
    In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
  • Conventional audio-visual models have independent audio and video branches.
  • 25 Feb 2022 1 repository listed
    From the results on four datasets regarding the above three tasks, our method yields remarkable performance improvements compared with the baselines, demonstrating its superiority on reducing the modality bias problem.
  • 1 Jan 2022 1 repository listed
    To the best of our knowledge, this is the first work to jointly model both feature and modality variation for different samples to provide trustworthy fusion in multi-modal classification.
  • 20 May 2021 1 repository listed
    In this work, we make two major contributions considering the above limitations: (1) we develop a Hindi-English code-mixed dataset, MaSaC, for the multi-modal sarcasm detection and humor classification in conversational…
  • 19 Sep 2019 1 repository listed
    Compared to natural images, understanding scientific figures is particularly hard for machines.
  • 3 Oct 2018 1 repository listed
    To learn feature representations of resulting images, standard Convolutional Neural Networks (CNNs) are employed for the classification task.
  • Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework
    9 Jun 2025 0 repositories listed
    Deepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation.
  • A Survey on Training-free Open-Vocabulary Semantic Segmentation
    28 May 2025 0 repositories listed
    Semantic segmentation is one of the most fundamental tasks in image understanding with a long history of research, and subsequently a myriad of different approaches.
  • A Comparative Study of Human Activity Recognition: Motion, Tactile, and multi-modal Approaches
    13 May 2025 0 repositories listed
    Human activity recognition (HAR) is essential for effective Human-Robot Collaboration (HRC), enabling robots to interpret and respond to human actions.
  • Multi-modal classification of forest biodiversity potential from 2D orthophotos and 3D airborne laser scanning point clouds
    3 Jan 2025 0 repositories listed
    Accurate assessment of forest biodiversity is crucial for ecosystem management and conservation.
  • Turbo your multi-modal classification with contrastive learning
    14 Sep 2024 0 repositories listed
    Contrastive learning has become one of the most impressive approaches for multi-modal representation learning.
  • FungiTastic: A multi-modal dataset and benchmark for image categorization
    24 Aug 2024 0 repositories listed
    We introduce a new, challenging benchmark and a dataset, FungiTastic, based on fungal records continuously collected over a twenty-year span.
  • Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images
    31 May 2024 0 repositories listed
    Vision-language models have emerged as a powerful tool for previously challenging multi-modal classification problem in the medical domain.
  • Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification
    7 Dec 2023 0 repositories listed
    Most convolutional neural network (CNN) based methods for skin cancer classification obtain their results using only dermatological images.
  • 22 Sep 2022 0 repositories listed
    Action recognition is an essential field for video understanding.
  • 22 Sep 2022 0 repositories listed
    While action recognition has been an active research area for several years, most existing approaches merely leverage the video modality as opposed to humans that efficiently process video and audio cues simultaneously.
  • Multi-Modal Hypergraph Diffusion Network with Dual Prior for Alzheimer Classification
    4 Apr 2022 0 repositories listed
    The automatic early diagnosis of prodromal stages of Alzheimer's disease is of great relevance for patient treatment to improve quality of life.
  • Multi Task Learning based Framework for Multimodal Classification
    1 Jun 2021 0 repositories listed
    Large-scale multi-modal classification aim to distinguish between different multi-modal data, and it has drawn dramatically attentions since last decade.
  • Cross-Modal Retrieval Augmentation for Multi-Modal Classification
    16 Apr 2021 0 repositories listed
    Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
  • Learning Multi-Modal Nonlinear Embeddings: Performance Bounds and an Algorithm
    3 Jun 2020 0 repositories listed
    While many approaches exist in the literature to learn low-dimensional representations for data collections in multiple modalities, the generalizability of multi-modal nonlinear embeddings to previously unseen data is a…
  • Assessing Post Deletion in Sina Weibo: Multi-modal Classification of Hot Topics
    26 Jun 2019 0 repositories listed
    Widespread Chinese social media applications such as Weibo are widely known for monitoring and deleting posts to conform to Chinese government requirements.
  • CuisineNet: Food Attributes Classification using Multi-scale Convolution Network
    30 May 2018 0 repositories listed
    Diversity of food and its attributes represents the culinary habits of peoples from different countries.
  • Efficient Large-Scale Multi-Modal Classification
    6 Feb 2018 0 repositories listed
    While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal.
  • Deep Multi-Modal Classification of Intraductal Papillary Mucinous Neoplasms (IPMN) with Canonical Correlation Analysis
    26 Oct 2017 0 repositories listed
    Pancreatic cancer has the poorest prognosis among all cancer types.
  • Multi-modal Fusion for Diabetes Mellitus and Impaired Glucose Regulation Detection
    12 Apr 2016 0 repositories listed

Syntology lines on 4 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections