Papers › MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

20 Apr 2023arXiv:2304.10592archive 2025-07-28

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

vision-cair/minigpt-4 officialmentioned in papermentioned on GitHubpytorchBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (arXiv:2310.09478; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
zyang1580/binllm mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Text-like Encoding of Collaborative Information in Large Language Models for Recommendation (arXiv:2406.03210; that paper's own run record: 14 ran (of which 2 constructed an object rather than computing a result; 9 with no instrument failure: 1 honoured, 1 violated, 7 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
2023-MindSpore-1/ms-code-16 mindsporeSyntology: not harvested report
2023-MindSpore-1/ms-code-224 mindsporeSyntology: not harvested report
2024-MindSpore-1/Code5 mindsporeSyntology: not harvested report
2024-MindSpore-1/Code6 mindsporeSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Language ModellingLarge Language ModelSpatial ReasoningVideo Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Visual Reasoning

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Spatial Reasoning EmbSpatial-Bench MiniGPT4 Generation 23.54 #5 of 5 Archive leaderboard report
Video Question Answering MVBench MiniGPT4 Avg. 18.8 #22 of 22 Archive leaderboard report
Visual Question Answering BenchLMM MiniGPT4-13B GPT-3.5 score 34.93 #9 of 10 Archive leaderboard report
Visual Question Answering MM-Vet MiniGPT-4-14B GPT-4 score 24.4±0.4 #224 of 231 Archive leaderboard report
Visual Question Answering MM-Vet MiniGPT-4-14B Params 14B #224 of 231 Archive leaderboard report
Visual Question Answering MM-Vet MiniGPT-4-8B GPT-4 score 22.1±0.1 #227 of 231 Archive leaderboard report
Visual Question Answering MM-Vet MiniGPT-4-8B Params 8B #227 of 231 Archive leaderboard report
Visual Question Answering (VQA) AutoHallusion miniGPT4 Overall Accuracy 51.0 #3 of 5 Archive leaderboard report
Visual Question Answering (VQA) InfiMM-Eval MiniGPT-v2 Abductive 13.28 #13 of 14 Archive leaderboard report
Visual Question Answering (VQA) InfiMM-Eval MiniGPT-v2 Analogical 5.69 #13 of 14 Archive leaderboard report
Visual Question Answering (VQA) InfiMM-Eval MiniGPT-v2 Deductive 11.02 #13 of 14 Archive leaderboard report
Visual Question Answering (VQA) InfiMM-Eval MiniGPT-v2 Overall score 10.43 #13 of 14 Archive leaderboard report
Visual Question Answering (VQA) InfiMM-Eval MiniGPT-v2 Params 8B #13 of 14 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPEDense ConnectionsDropoutGPT-4Label SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections