Browse State-of-the-Art › multimodal generation
multimodal generation
53 papers with code · 1 benchmark · 6 datasets archive 2025-07-28
Multimodal generation refers to the process of generating outputs that incorporate multiple modalities, such as images, text, and sound. This can be done using deep learning models that are trained on data that includes multiple modalities, allowing the models to generate output that is informed by more than one type of data.
For example, a multimodal generation model could be trained to generate captions for images that incorporate both text and visual information. The model could learn to identify objects in the image and generate descriptions of them in natural language, while also taking into account contextual information and the relationships between the objects in the image.
Multimodal generation can also be used in other applications, such as generating realistic images from textual descriptions or generating audio descriptions of video content. By combining multiple modalities in this way, multimodal generation models can produce more accurate and comprehensive output, making them useful for a wide range of applications.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| Multi-Modal CelebA-HQ (1 row) | Diffusion | Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
6 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 98 · with a code link: 53 · where Syntology ran a sample: 18 (16 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (18 of 98 tagged: 16 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 98 of 98, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
7 Apr 2024 3 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)Such user preferences are then fed into a generator, such as a multimodal LLM or diffusion model, to produce personalized content.
-
29 Feb 2024 3 repositories listedWe first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators.
-
27 Sep 2023 3 repositories listed Syntology community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)Each dimension is quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets.
-
20 May 2025 2 repositories listed Syntology 19 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 4 honoured, 0 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified (of 21 harvested samples) · 3 pointer-only (licence)Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems.
-
27 Mar 2025 2 repositories listedVision-to-music Generation, including video-to-music and image-to-music tasks, is a significant branch of multimodal artificial intelligence demonstrating vast application prospects in fields such as film scoring, short…
-
10 Mar 2023 2 repositories listedThis stream is subsequently fed into the decoder-based transformer to generate visual re-creations and textual feedback in the second stage.
-
31 Jan 2023 2 repositories listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples) · 1 pointer-only (licence)We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved…
-
11 Jun 2021 2 repositories listedThis adversarial loss guarantees the map is diverse -- a very wide range of anime can be produced from a single content code.
-
23 Jun 2025 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)To facilitate the training of OmniGen2, we developed comprehensive data construction pipelines, encompassing image editing and in-context generation data.
-
29 May 2025 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)Unlike prior unified diffusion models trained from scratch, Muddit integrates strong visual priors from a pretrained text-to-image backbone with a lightweight text decoder, enabling flexible and high-quality multimodal…
-
24 May 2025 1 repository listedRecent breakthroughs in large multimodal models (LMMs), such as the impressive GPT-4o-Native, have demonstrated remarkable proficiency in following general-purpose instructions for image generation.
-
8 Apr 2025 1 repository listedThe landscape of image generation has rapidly evolved, from early GAN-based approaches to diffusion models and, most recently, to unified generative architectures that seek to bridge understanding and generation tasks.
-
3 Apr 2025 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 2 pointer-only (licence)Reinforcement fine-tuning has instrumental enhanced the instruction-following and reasoning abilities of large language models.
-
18 Mar 2025 1 repository listedWorld models significantly enhance hierarchical understanding, improving data integration and learning efficiency.
-
16 Mar 2025 1 repository listedBy extending the advantage of chain-of-thought (CoT) reasoning in human-like step-by-step processes to multimodal contexts, multimodal CoT (MCoT) reasoning has recently garnered significant research attention…
-
11 Mar 2025 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 7 where Syntology's instrument failed) · 1 unverified (of 8 harvested samples) · 4 pointer-only (licence)The model fully leverages Mamba-2's high computational and memory efficiency, extending its capabilities from text generation to multimodal generation.
-
7 Mar 2025 1 repository listedRecent advances in human preference alignment have significantly enhanced multimodal generation and understanding.
-
3 Mar 2025 1 repository listedIn this work, we introduce WeGen, a model that unifies multimodal generation and understanding, and promotes their interplay in iterative generation.
-
12 Feb 2025 1 repository listedLarge Language Models (LLMs) struggle with hallucinations and outdated knowledge due to their reliance on static training data.
-
8 Feb 2025 1 repository listedHowever, the key challenge is establishing a unified denoising perspective for both image and text generation, which is essential for establishing the consistency mapping.
-
6 Feb 2025 1 repository listedTo address this, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which we aim to generate multimodal answers that combine both text and images, fully leveraging the multimodal data…
-
23 Dec 2024 1 repository listedIn Artificial Intelligence Generated Content (AIGC), distinguishing AI-synthesized images from natural ones remains a key challenge.
-
11 Dec 2024 1 repository listedIn this work, we propose Latent Language Modeling (LatentLM), which seamlessly integrates continuous and discrete data using causal Transformers.
-
27 Nov 2024 1 repository listedWhile the progress in unified models offers new solutions, existing benchmarks are insufficient for evaluating these methods due to limitations in data size and diversity.
-
25 Nov 2024 1 repository listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)This paper investigates an intriguing task of Multi-modal Retrieval Augmented Multi-modal Generation (M²RAG).
-
15 Oct 2024 1 repository listedAs one of the most popular and sought-after generative models in the recent years, diffusion models have sparked the interests of many researchers and steadily shown excellent advantage in various generative tasks such…
-
17 Sep 2024 1 repository listedIn this paper, we propose a practical framework - MM2Latent - for multimodal image generation and editing.
-
16 Sep 2024 1 repository listedThis report presents PixelBytes, an approach for unified multimodal representation learning.
-
3 Sep 2024 1 repository listedThis report introduces PixelBytes Embedding, a novel approach for unified multimodal representation learning.
-
21 Aug 2024 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 11 pointer-only (licence)In this work, we present UniFashion, a unified framework that simultaneously tackles the challenges of multimodal generation and retrieval tasks within the fashion domain, integrating image generation with retrieval…
-
5 Aug 2024 1 repository listed
-
30 Jul 2024 1 repository listed
-
23 Jul 2024 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 7 unverified (of 18 harvested samples)
-
ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation8 Jul 2024 1 repository listed
-
1 Jul 2024 1 repository listed
-
13 Jun 2024 1 repository listed
-
29 May 2024 1 repository listed
-
13 Oct 2023 1 repository listed
-
3 Oct 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples) · 1 pointer-only (licence)
-
2 Oct 2023 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 2 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 11 pointer-only (licence)
-
20 Sep 2023 1 repository listed Syntology official (archive's flag): 1 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)
-
16 Aug 2023 1 repository listed Syntology official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 4 honoured, 4 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 17 harvested samples) · 17 pointer-only (licence)
-
7 Jun 2023 1 repository listed
-
26 May 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 2 pointer-only (licence)
-
24 May 2023 1 repository listed
-
24 May 2023 1 repository listed
-
6 Jan 2023 1 repository listed
-
1 Dec 2022 1 repository listed
-
27 Nov 2022 1 repository listed
-
25 Aug 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples) · 1 pointer-only (licence)
-
3 Aug 2022 1 repository listed
-
15 Apr 2022 1 repository listed
-
12 Jun 2019 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 1 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 12 harvested samples)
-
PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis22 Jun 2025 0 repositories listed
-
Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation12 Jun 2025 0 repositories listed
-
MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation9 Jun 2025 0 repositories listed
-
MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask Multimodal Generation23 May 2025 0 repositories listed
-
Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models20 May 2025 0 repositories listed
-
Preliminary Explorations with GPT-4o(mni) Native Image Generation6 May 2025 0 repositories listed
-
Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability9 Apr 2025 0 repositories listed
-
Stance-Driven Multimodal Controlled Statement Generation: New Dataset and Task4 Apr 2025 0 repositories listed
-
Generating Multimodal Driving Scenes via Next-Scene Prediction19 Mar 2025 0 repositories listed
-
ProtTeX: Structure-In-Context Reasoning and Editing of Proteins with Large Language Models11 Mar 2025 0 repositories listed
-
ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model with Interleaved Multimodal Generation via Asymmetric Synergy9 Mar 2025 0 repositories listed
-
Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction8 Mar 2025 0 repositories listed
-
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval18 Feb 2025 0 repositories listed
-
A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond17 Feb 2025 0 repositories listed
-
Artificial Intelligence in Creative Industries: Advances Prior to 20256 Jan 2025 0 repositories listed
-
RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction24 Dec 2024 0 repositories listed
-
LMFusion: Adapting Pretrained Language Models for Multimodal Generation19 Dec 2024 0 repositories listed
-
Visatronic: A Multimodal Decoder-Only Model for Speech Synthesis26 Nov 2024 0 repositories listed
-
Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains22 Nov 2024 0 repositories listed
-
A Survey on Vision Autoregressive Model13 Nov 2024 0 repositories listed
-
A Survey of Emerging Approaches and Advances in Video Generation9 Nov 2024 0 repositories listed
-
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation14 Oct 2024 0 repositories listed
-
ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction7 Oct 2024 0 repositories listed
-
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference30 Sep 2024 0 repositories listed
-
Multimodal ELBO with Diffusion Decoders29 Aug 2024 0 repositories listed
-
Learning Multimodal Latent Space with EBM Prior and MCMC Inference20 Aug 2024 0 repositories listed
-
Diffusion Models For Multi-Modal Generative Modeling24 Jul 2024 0 repositories listed
-
The Evolution of Multimodal Model Architectures28 May 2024 0 repositories listed
-
C3LLM: Conditional Multimodal Content Generation Using Large Language Models25 May 2024 0 repositories listed
-
Multimodal Pretraining and Generation for Recommendation: A Tutorial11 May 2024 0 repositories listed
-
PCQA: A Strong Baseline for AIGC Quality Assessment Based on Prompt Condition20 Apr 2024 0 repositories listed
-
Food Development through Co-creation with AI: bread with a "taste of love"19 Apr 2024 0 repositories listed
-
3D-VLA: A 3D Vision-Language-Action Generative World Model14 Mar 2024 0 repositories listed
-
CLIP Model for Images to Textual Prompts Based on Top-k Neighbors18 Jan 2024 0 repositories listed
-
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov 2023 0 repositories listed
-
C3Net: Compound Conditioned ControlNet for Multimodal Content Generation29 Nov 2023 0 repositories listed
-
Towards Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs27 Nov 2023 0 repositories listed
-
LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts1 Oct 2023 0 repositories listed
-
Consistent Multimodal Generation via A Unified GAN Framework4 Jul 2023 0 repositories listed
-
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs30 Jun 2023 0 repositories listed
-
DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion19 May 2023 0 repositories listed
-
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation12 Mar 2022 0 repositories listed
-
16 Nov 2021 0 repositories listed
-
I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion17 Feb 2021 0 repositories listed
-
Latent Dirichlet Allocation in Generative Adversarial Networks17 Dec 2018 0 repositories listed
Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections