Browse State-of-the-Art › Multi-modal Classification
Multi-modal Classification
12 papers with code · 2 benchmarks · 4 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| VGG-Sound (4 rows) | MMT | Multiscale Multimodal Transformer for Multimodal Action Recognition | — | — | Compare |
| AudioSet (2 rows) | CAV-MAE | Contrastive Audio-Visual Masked Autoencoder | code | Syntology official: harvested, nothing ran · 0 ran · 1 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
4 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 31 · with a code link: 12 · where Syntology ran a sample: 3 (2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (3 of 31 tagged: 2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 31 of 31, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
29 May 2019 3 repositories listed Syntology 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples)Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal…
-
23 Dec 2024 1 repository listedMultimodal AI models are increasingly used in fields like healthcare, finance, and autonomous driving, where information is drawn from multiple sources or modalities such as images, texts, audios, videos.
-
13 Nov 2024 1 repository listedThe prevalence of multi-modal content on social media complicates automated moderation strategies.
-
27 Jul 2023 1 repository listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)In a joint vision-language space, a text feature (e. g., from "a photo of a dog") could effectively represent its relevant image features (e. g., from dog photos).
-
4 Mar 2023 1 repository listedIn the fashion domain, there exists a variety of vision-and-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning.
-
2 Oct 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
-
29 Jul 2022 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 1 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (of 14 harvested samples) · 3 pointer-only (licence)Conventional audio-visual models have independent audio and video branches.
-
25 Feb 2022 1 repository listedFrom the results on four datasets regarding the above three tasks, our method yields remarkable performance improvements compared with the baselines, demonstrating its superiority on reducing the modality bias problem.
-
1 Jan 2022 1 repository listedTo the best of our knowledge, this is the first work to jointly model both feature and modality variation for different samples to provide trustworthy fusion in multi-modal classification.
-
20 May 2021 1 repository listedIn this work, we make two major contributions considering the above limitations: (1) we develop a Hindi-English code-mixed dataset, MaSaC, for the multi-modal sarcasm detection and humor classification in conversational…
-
19 Sep 2019 1 repository listedCompared to natural images, understanding scientific figures is particularly hard for machines.
-
3 Oct 2018 1 repository listedTo learn feature representations of resulting images, standard Convolutional Neural Networks (CNNs) are employed for the classification task.
-
Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework9 Jun 2025 0 repositories listedDeepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation.
-
A Survey on Training-free Open-Vocabulary Semantic Segmentation28 May 2025 0 repositories listedSemantic segmentation is one of the most fundamental tasks in image understanding with a long history of research, and subsequently a myriad of different approaches.
-
A Comparative Study of Human Activity Recognition: Motion, Tactile, and multi-modal Approaches13 May 2025 0 repositories listedHuman activity recognition (HAR) is essential for effective Human-Robot Collaboration (HRC), enabling robots to interpret and respond to human actions.
-
Multi-modal classification of forest biodiversity potential from 2D orthophotos and 3D airborne laser scanning point clouds3 Jan 2025 0 repositories listedAccurate assessment of forest biodiversity is crucial for ecosystem management and conservation.
-
Turbo your multi-modal classification with contrastive learning14 Sep 2024 0 repositories listedContrastive learning has become one of the most impressive approaches for multi-modal representation learning.
-
FungiTastic: A multi-modal dataset and benchmark for image categorization24 Aug 2024 0 repositories listedWe introduce a new, challenging benchmark and a dataset, FungiTastic, based on fungal records continuously collected over a twenty-year span.
-
Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images31 May 2024 0 repositories listedVision-language models have emerged as a powerful tool for previously challenging multi-modal classification problem in the medical domain.
-
Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification7 Dec 2023 0 repositories listedMost convolutional neural network (CNN) based methods for skin cancer classification obtain their results using only dermatological images.
-
22 Sep 2022 0 repositories listedAction recognition is an essential field for video understanding.
-
22 Sep 2022 0 repositories listedWhile action recognition has been an active research area for several years, most existing approaches merely leverage the video modality as opposed to humans that efficiently process video and audio cues simultaneously.
-
Multi-Modal Hypergraph Diffusion Network with Dual Prior for Alzheimer Classification4 Apr 2022 0 repositories listedThe automatic early diagnosis of prodromal stages of Alzheimer's disease is of great relevance for patient treatment to improve quality of life.
-
Multi Task Learning based Framework for Multimodal Classification1 Jun 2021 0 repositories listedLarge-scale multi-modal classification aim to distinguish between different multi-modal data, and it has drawn dramatically attentions since last decade.
-
Cross-Modal Retrieval Augmentation for Multi-Modal Classification16 Apr 2021 0 repositories listedRecent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
-
Learning Multi-Modal Nonlinear Embeddings: Performance Bounds and an Algorithm3 Jun 2020 0 repositories listedWhile many approaches exist in the literature to learn low-dimensional representations for data collections in multiple modalities, the generalizability of multi-modal nonlinear embeddings to previously unseen data is a…
-
Assessing Post Deletion in Sina Weibo: Multi-modal Classification of Hot Topics26 Jun 2019 0 repositories listedWidespread Chinese social media applications such as Weibo are widely known for monitoring and deleting posts to conform to Chinese government requirements.
-
CuisineNet: Food Attributes Classification using Multi-scale Convolution Network30 May 2018 0 repositories listedDiversity of food and its attributes represents the culinary habits of peoples from different countries.
-
Efficient Large-Scale Multi-Modal Classification6 Feb 2018 0 repositories listedWhile the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal.
-
Deep Multi-Modal Classification of Intraductal Papillary Mucinous Neoplasms (IPMN) with Canonical Correlation Analysis26 Oct 2017 0 repositories listedPancreatic cancer has the poorest prognosis among all cancer types.
-
Multi-modal Fusion for Diabetes Mellitus and Impaired Glucose Regulation Detection12 Apr 2016 0 repositories listed
Syntology lines on 4 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections