Browse State-of-the-Art › multimodal interaction
multimodal interaction
41 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 106 · with a code link: 41 · where Syntology ran a sample: 9 (8 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (9 of 106 tagged: 8 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 106, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
5 Feb 2021 6 repositories listed Syntology official: harvested, nothing ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified; the one sample that ran constructed an object rather than computing a result (of 4 harvested samples) · 1 pointer-only (licence)Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
-
16 Jun 2025 2 repositories listedThe emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction.
-
13 Oct 2023 2 repositories listedConsequently, our work complements research on the performance of MLLMs in multimodal comprehension tasks, achieving a more comprehensive and holistic evaluation of MLLMs.
-
25 May 2025 1 repository listedCharts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis.
-
25 May 2025 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (of 4 harvested samples) · 4 pointer-only (licence)Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources.
-
5 May 2025 1 repository listedWe introduce Ming-Lite-Uni, an open-source multimodal framework featuring a newly designed unified visual generator and a native multimodal autoregressive model tailored for unifying vision and language.
-
13 Mar 2025 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 1 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 12 harvested samples) · 2 pointer-only (licence)Open-vocabulary multiple object tracking aims to generalize trackers to unseen categories during training, enabling their application across a variety of real-world scenarios.
-
27 Jan 2025 1 repository listedTo tackle the EMDRC task, we construct a new dataset based on an existing MDRC dataset.
-
14 Nov 2024 1 repository listedTo address this limitation, we introduce Spider, a novel efficient Any-to-Many Modalities Generation (AMMG) framework, which can generate an arbitrary combination of modalities 'Text + Xs', such as Text + {Image and…
-
13 Nov 2024 1 repository listedRecent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment.
-
4 Nov 2024 1 repository listedThis survey provides a comprehensive review of mobile agent technologies, focusing on recent advancements that enhance real-time adaptability and multimodal interaction.
-
26 Oct 2024 1 repository listedMultimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding.
-
7 Oct 2024 1 repository listedJoint communication and sensing (JC&S) is emerging as a key component in 5G and 6G networks, enabling dynamic adaptation to environmental changes and enhancing contextual awareness for optimized communication.
-
8 Sep 2024 1 repository listedMaTAV is with the advantages of aligning unimodal features to ensure consistency across different modalities and handling long input sequences to better capture contextual multimodal information.
-
2 Aug 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)Large multimodal models (LMMs) excel in adhering to human instructions.
-
23 Jul 2024 1 repository listedTo this end, we propose UniMEL, a unified framework which establishes a new paradigm to process multimodal entity linking tasks using LLMs.
-
10 Jul 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 1 pointer-only (licence)Moreover, TIP proposes a versatile tabular encoder tailored for incomplete, heterogeneous tabular data and a multimodal interaction module for inter-modality representation learning.
-
1 Jul 2024 1 repository listedWe introduce the concept of "empathic grounding" in conversational agents as an extension of Clark's conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the…
-
30 Jun 2024 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 2 unverified (of 2 harvested samples) · 2 pointer-only (licence)To mitigate this, we further propose a training-free method called Residual Visual Decoding, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with…
-
26 Jun 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples)Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges…
-
22 Apr 2024 1 repository listedNAE is a more challenging task because it requires both narrative flexibility and evaluation rigor.
-
19 Apr 2024 1 repository listedIn this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment…
-
29 Mar 2024 1 repository listedIn this paper, we introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting.
-
7 Jan 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 2 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 7 pointer-only (licence)To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can…
-
16 Nov 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled.
-
26 Oct 2023 1 repository listedSimulation is an invaluable tool for developing and evaluating controllers for self-driving cars.
-
28 Jul 2023 1 repository listedRUME is applied to extract conversation-level contextual emotional cues while pulling together data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM…
-
19 Jul 2023 1 repository listedThen, to derive the similarity matching score for each mention-entity pair, we device three interaction units to comprehensively explore the intra-modal interaction and inter-modal fusion among features of entities and…
-
1 Jul 2023 1 repository listedWith the extracted face sequences, we propose a multimodal facial expression-aware emotion recognition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion…
-
22 Jun 2023 1 repository listedMultimodal Entity Linking (MEL) is the task of mapping mentions with multimodal contexts to the referent entities from a knowledge base.
-
11 May 2023 1 repository listed
-
22 Mar 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 3 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 9 harvested samples)
-
7 Mar 2023 1 repository listed
-
7 Nov 2022 1 repository listed
-
9 May 2022 1 repository listed
-
3 Nov 2021 1 repository listed
-
1 Nov 2021 1 repository listed
-
10 Sep 2021 1 repository listed
-
11 Jul 2021 1 repository listed
-
1 Jul 2020 1 repository listed
-
23 Mar 2017 1 repository listed
-
A multi-stage augmented multimodal interaction network for fish feeding intensity quantification17 Jun 2025 0 repositories listed
-
InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback29 May 2025 0 repositories listed
-
DeepSORT-Driven Visual Tracking Approach for Gesture Recognition in Interactive Systems11 May 2025 0 repositories listed
-
A Survey of Interactive Generative Video30 Apr 2025 0 repositories listed
-
Immersive Multimedia Communication: State-of-the-Art on eXtended Reality Streaming27 Mar 2025 0 repositories listed
-
ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting20 Feb 2025 0 repositories listed
-
Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving12 Feb 2025 0 repositories listed
-
FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection8 Jan 2025 0 repositories listed
-
MODfinity: Unsupervised Domain Adaptation with Multimodal Information Flow Intertwining1 Jan 2025 0 repositories listed
-
Computer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer24 Dec 2024 0 repositories listed
-
CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation15 Nov 2024 0 repositories listed
-
Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability15 Nov 2024 0 repositories listed
-
Phase Diagram of Vision Large Language Models Inference: A Perspective from Interaction across Image and Instruction1 Nov 2024 0 repositories listed
-
Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes29 Oct 2024 0 repositories listed
-
Retrospective Learning from Interactions17 Oct 2024 0 repositories listed
-
Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant30 Sep 2024 0 repositories listed
-
LLM-Assisted Visual Analytics: Opportunities and Challenges4 Sep 2024 0 repositories listed
-
16 Aug 2024 0 repositories listed
-
A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models25 Jul 2024 0 repositories listed
-
Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic25 Jul 2024 0 repositories listed
-
HGNET: A Hierarchical Feature Guided Network for Occupancy Flow Field Prediction1 Jul 2024 0 repositories listed
-
A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)27 Jun 2024 0 repositories listed
-
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents27 Jun 2024 0 repositories listed
-
EMMI -- Empathic Multimodal Motivational Interviews Dataset: Analyses and Annotations24 Jun 2024 0 repositories listed
-
27 Apr 2024 0 repositories listed
-
BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AI20 Mar 2024 0 repositories listed
-
Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction16 Mar 2024 0 repositories listed
-
On the Arrow of Inference22 Feb 2024 0 repositories listed
-
Memory-Inspired Temporal Prompt Interaction for Text-Image Classification26 Jan 2024 0 repositories listed
-
Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition20 Sep 2023 0 repositories listed
-
Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems11 Sep 2023 0 repositories listed
-
Expanding the Role of Affective Phenomena in Multimodal Interaction Research18 May 2023 0 repositories listed
-
1 Jan 2023 0 repositories listed
-
InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis20 Dec 2022 0 repositories listed
-
A novel multimodal dynamic fusion network for disfluency detection in spoken utterances27 Nov 2022 0 repositories listed
-
Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces7 Nov 2022 0 repositories listed
-
Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment10 Oct 2022 0 repositories listed
-
MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning9 Oct 2022 0 repositories listed
-
On the Horizon: Interactive and Compositional Deepfakes5 Sep 2022 0 repositories listed
-
Chat-to-Design: AI Assisted Personalized Fashion Design3 Jul 2022 0 repositories listed
-
RoomReader: A Multimodal Corpus of Online Multiparty Conversational Interactions1 Jun 2022 0 repositories listed
-
The VoxWorld Platform for Multimodal Embodied Agents1 Jun 2022 0 repositories listed
-
Graph-based Fine-grained Multimodal Attention Mechanism for Sentiment Analysis16 Nov 2021 0 repositories listed
-
Neural dSCA: demixing multimodal interaction among brain areas during naturalistic experiments5 Jun 2021 0 repositories listed
-
SocialInteractionGAN: Multi-person Interaction Sequence Generation10 Mar 2021 0 repositories listed
-
Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset20 Aug 2020 0 repositories listed
-
SBAT: Video Captioning with Sparse Boundary-Aware Transformer23 Jul 2020 0 repositories listed
-
Dual Convolutional LSTM Network for Referring Image Segmentation30 Jan 2020 0 repositories listed
-
Scene Graph based Image Retrieval -- A case study on the CLEVR Dataset3 Nov 2019 0 repositories listed
-
Corpus of Multimodal Interaction for Collaborative Planning1 Jun 2019 0 repositories listed
-
HUMBO: Bridging Response Generation and Facial Expression Synthesis24 May 2019 0 repositories listed
-
Guidelines for creating man-machine multimodal interfaces29 Jan 2019 0 repositories listed
-
Shaping a social robot's humor with Natural Language Generation and socially-aware reinforcement learning1 Nov 2018 0 repositories listed
-
EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction23 Aug 2018 0 repositories listed
-
21 Aug 2018 0 repositories listed
-
Toward Multimodal Interaction in Scalable Visual Digital Evidence Visualization Using Computer Vision Techniques and ISS1 Aug 2018 0 repositories listed
-
An Evaluation Framework for Multimodal Interaction1 May 2018 0 repositories listed
-
Automatized Generation of Alphabets of Symbols16 Jul 2017 0 repositories listed
-
From Modal to Multimodal Ambiguities: a Classification Approach4 Apr 2017 0 repositories listed
Syntology lines on 10 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections