Papers › Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

10 May 2023NeurIPS 2023 11arXiv:2305.06324archive 2025-07-28

Hassan Akbari, Dan Kondratyuk, Yin Cui, Rachel Hornung, Huisheng Wang, Hartwig Adam

We present Integrated Multimodal Perception (IMP), a simple and scalable multimodal multi-task training and modeling approach. IMP integrates multimodal inputs including image, video, text, and audio into a single Transformer encoder with minimal modality-specific components. IMP makes use of a novel design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) for efficient model and task scaling. We conduct extensive empirical studies and reveal the following key insights: 1) Performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model. 2) Sparsification with MoE on a single modality-agnostic encoder substantially improves the performance, outperforming dense models that use modality-specific encoders or additional fusion layers and greatly mitigates the conflicts between modalities. IMP achieves competitive performance on a wide range of downstream tasks including video classification, image classification, image-text, and video-text retrieval. Most notably, we train a sparse IMP-MoE-L variant focusing on video tasks that achieves new state-of-the-art in zero-shot video classification: 77.0% on Kinetics-400, 76.8% on Kinetics-600, and 68.3% on Kinetics-700, improving the previous state-of-the-art by +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total training computational cost.

Paper PDF Conference PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Classification Image Classification Mixture-of-Experts Text Retrieval Video Classification Video-Text Retrieval Zero-Shot Action Recognition Zero-Shot Environment Sound Classification Zero-Shot Learning Zero-Shot Transfer Image Classification image-classification

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Zero-Shot Action Recognition	HMDB51	IMP-MoE-L	Top-1 Accuracy	59.1	#5 of 29	Archive leaderboard	report
Zero-Shot Action Recognition	Kinetics	IMP-MoE-L	Top-1 Accuracy	76.8	#2 of 20	Archive leaderboard	report
Zero-Shot Action Recognition	UCF101	IMP-MoE-L	Top-1 Accuracy	91.5	#2 of 35	Archive leaderboard	report
Zero-Shot Transfer Image Classification	ImageNet	IMP-MoE-L	Accuracy (Private)	83.9	#8 of 23	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections