multimodal interaction

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 41 papers with code (106 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
  5 Feb 2021 6 repositories listed Syntology ran 1 of 4 samples · 3 unverified · 1 pointer-only (licence)
  Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
• Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
  16 Jun 2025 2 repositories listed
  The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction.
• MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  13 Oct 2023 2 repositories listed
  Consequently, our work complements research on the performance of MLLMs in multimodal comprehension tasks, achieving a more comprehensive and holistic evaluation of MLLMs.
• ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
  25 May 2025 1 repository listed
  Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis.
• I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
  25 May 2025 1 repository listed Syntology ran 4 of 4 samples · 0 unverified · 4 pointer-only (licence)
  Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources.
• Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction
  5 May 2025 1 repository listed
  We introduce Ming-Lite-Uni, an open-source multimodal framework featuring a newly designed unified visual generator and a native multimodal autoregressive model tailored for unifying vision and language.
• OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer
  13 Mar 2025 1 repository listed Syntology ran 9 of 12 samples · 3 unverified
  Open-vocabulary multiple object tracking aims to generalize trackers to unseen categories during training, enabling their application across a variety of real-world scenarios.
• Towards Explainable Multimodal Depression Recognition for Clinical Interviews
  27 Jan 2025 1 repository listed
  To tackle the EMDRC task, we construct a new dataset based on an existing MDRC dataset.
• Spider: Any-to-Many Multimodal LLM
  14 Nov 2024 1 repository listed
  To address this limitation, we introduce Spider, a novel efficient Any-to-Many Modalities Generation (AMMG) framework, which can generate an arbitrary combination of modalities 'Text + Xs', such as Text + {Image and…
• MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval
  13 Nov 2024 1 repository listed
  Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment.
• Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
  4 Nov 2024 1 repository listed
  This survey provides a comprehensive review of mobile agent technologies, focusing on recent advancements that enhance real-time adaptability and multimodal interaction.
• LLMs Can Evolve Continually on Modality for X-Modal Reasoning
  26 Oct 2024 1 repository listed
  Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding.
• Spatio-Temporal 3D Point Clouds from WiFi-CSI Data via Transformer Networks
  7 Oct 2024 1 repository listed
  Joint communication and sensing (JC\&S) is emerging as a key component in 5G and 6G networks, enabling dynamic adaptation to environmental changes and enhancing contextual awareness for optimized communication.
• Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations
  8 Sep 2024 1 repository listed
  MaTAV is with the advantages of aligning unimodal features to ensure consistency across different modalities and handling long input sequences to better capture contextual multimodal information.
• Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions
  2 Aug 2024 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Large multimodal models (LMMs) excel in adhering to human instructions.
• UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models
  23 Jul 2024 1 repository listed
  To this end, we propose UniMEL, a unified framework which establishes a new paradigm to process multimodal entity linking tasks using LLMs.
• TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data
  10 Jul 2024 1 repository listed Syntology ran 2 of 4 samples · 2 unverified
  Moreover, TIP proposes a versatile tabular encoder tailored for incomplete, heterogeneous tabular data and a multimodal interaction module for inter-modality representation learning.
• Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents
  1 Jul 2024 1 repository listed
  We introduce the concept of "empathic grounding" in conversational agents as an extension of Clark's conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the…
• Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
  30 Jun 2024 1 repository listed Syntology ran 0 of 2 samples · 2 unverified · 2 pointer-only (licence)
  To mitigate this, we further propose a training-free method called Residual Visual Decoding, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with…
• LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
  26 Jun 2024 1 repository listed Syntology ran 4 of 5 samples · 1 unverified
  Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges…
• Narrative Action Evaluation with Prompt-Guided Multimodal Interaction
  22 Apr 2024 1 repository listed
  NAE is a more challenging task because it requires both narrative flexibility and evaluation rigor.
• Cooperative Sentiment Agents for Multimodal Sentiment Analysis
  19 Apr 2024 1 repository listed
  In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment…
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
  29 Mar 2024 1 repository listed
  In this paper, we introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting.
• Agent AI: Surveying the Horizons of Multimodal Interaction
  7 Jan 2024 1 repository listed Syntology ran 4 of 7 samples · 3 unverified · 7 pointer-only (licence)
  To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can…
• MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts
  16 Nov 2023 1 repository listed Syntology ran 4 of 5 samples · 1 unverified · 5 pointer-only (licence)
  Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled.
• Dialogue-based generation of self-driving simulation scenarios using Large Language Models
  26 Oct 2023 1 repository listed
  Simulation is an invaluable tool for developing and evaluating controllers for self-driving cars.
• CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition
  28 Jul 2023 1 repository listed
  RUME is applied to extract conversation-level contextual emotional cues while pulling together data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM…
• Multi-Grained Multimodal Interaction Network for Entity Linking
  19 Jul 2023 1 repository listed
  Then, to derive the similarity matching score for each mention-entity pair, we device three interaction units to comprehensively explore the intra-modal interaction and inter-modal fusion among features of entities and…
• A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations
  1 Jul 2023 1 repository listed
  With the extracted face sequences, we propose a multimodal facial expression-aware emotion recognition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion…
• Generative Multimodal Entity Linking
  22 Jun 2023 1 repository listed
  Multimodal Entity Linking (MEL) is the task of mapping mentions with multimodal contexts to the referent entities from a knowledge base.

Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.