Papers › Cross-modal RAG: Sub-dimensional Retrieval-Augmented Text-to-Image Generation

Cross-modal RAG: Sub-dimensional Retrieval-Augmented Text-to-Image Generation

28 May 2025arXiv:2505.21956archive 2025-07-28

Mengdan Zhu, Senhao Cheng, Guangji Bai, Yifei Zhang, Liang Zhao

Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in both retrieval and generation quality, while maintaining high efficiency.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

mengdanzhu/cross-modal-rag officialmentioned in paperApache-2.0 report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image Generation Language Modeling Language Modelling Large Language Model Multimodal Large Language Model RAG Retrieval Retrieval-augmented Generation Text to Image Generation Text-to-Image Generation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Adam Attention Attention Dropout BART BERT BPE Dense Connections Dropout Layer Normalization Linear Layer Linear Warmup With Linear Decay Multi-Head Attention RAG Residual Connection SET Softmax Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections