{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minigpt-v2-large-language-model-as-a-unified","title":"MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning","arxiv_id":"2310.09478","date":"2023-10-14","proceeding":null,"authors":["Jun Chen","Deyao Zhu","Xiaoqian Shen","Xiang Li","Zechun Liu","Pengchuan Zhang","Raghuraman Krishnamoorthi","Vikas Chandra","Yunyang Xiong","Mohamed Elhoseiny"],"abstract":"Large language models have shown their remarkable capabilities as a general interface for various language-related applications. Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others. The challenge is to use a single model for performing diverse vision-language tasks effectively with simple multi-modal instructions. Towards this objective, we introduce MiniGPT-v2, a model that can be treated as a unified interface for better handling various vision-language tasks. We propose using unique identifiers for different tasks when training the model. These identifiers enable our model to better distinguish each task instruction effortlessly and also improve the model learning efficiency for each task. After the three-stage training, the experimental results show that MiniGPT-v2 achieves strong performance on many visual question-answering and visual grounding benchmarks compared to other vision-language generalist models. Our model and codes are available at https://minigpt-v2.github.io/","url_abs":"https://arxiv.org/abs/2310.09478v3","url_pdf":"https://arxiv.org/pdf/2310.09478v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minigpt-v2-large-language-model-as-a-unified","repo_url":"https://github.com/vision-cair/minigpt-4","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"minigpt-v2-large-language-model-as-a-unified","repo_url":"https://github.com/zebangcheng/emotion-llama","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":null,"task_name":"Image Description"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"natural-language-visual-grounding","task_name":"Natural Language Visual Grounding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-generation","task_name":"Referring expression generation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-coloninst-v1-seen","task":"Image Classification","dataset":"ColonINST-v1 (Seen)","model":"MiniGPT-v2 (w/ LoRA, w/o extra data)","rank_in_archive_order":12,"of":17,"metrics":{"Accuray":"91.49"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-seen","task":"Image Classification","dataset":"ColonINST-v1 (Seen)","model":"MiniGPT-v2 (w/ LoRA, w/ extra data)","rank_in_archive_order":14,"of":17,"metrics":{"Accuray":"90.00"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-unseen","task":"Image Classification","dataset":"ColonINST-v1 (Unseen)","model":"MiniGPT-v2\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":11,"of":17,"metrics":{"Accuray":"77.93"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-coloninst-v1-unseen","task":"Image Classification","dataset":"ColonINST-v1 (Unseen)","model":"MiniGPT-v2\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":13,"of":17,"metrics":{"Accuray":"76.82"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-visual-grounding-on","task":"Natural Language Visual Grounding","dataset":"ScreenSpot","model":"MiniGPT-v2","rank_in_archive_order":16,"of":18,"metrics":{"Accuracy (%)":"5.7"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst","task":"Referring expression generation","dataset":"ColonINST-v1 (Seen)","model":"MiniGPT-v2\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":13,"of":17,"metrics":{"Accuray":"94.69"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst","task":"Referring expression generation","dataset":"ColonINST-v1 (Seen)","model":"MiniGPT-v2\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":15,"of":17,"metrics":{"Accuray":"87.65"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst-1","task":"Referring expression generation","dataset":"ColonINST-v1 (Unseen)","model":"MiniGPT-v2\n  (w/ LoRA, w/o extra data)","rank_in_archive_order":10,"of":17,"metrics":{"Accuray":"72.05"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-generation-on-coloninst-1","task":"Referring expression generation","dataset":"ColonINST-v1 (Unseen)","model":"MiniGPT-v2\n  (w/ LoRA, w/ extra data)","rank_in_archive_order":12,"of":17,"metrics":{"Accuray":"70.23"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-benchlmm","task":"Visual Question Answering","dataset":"BenchLMM","model":"MiniGPTv2-7B","rank_in_archive_order":10,"of":10,"metrics":{"GPT-3.5 score":"30.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.09478","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09478"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zebangcheng/emotion-llama","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vision-cair/minigpt-4","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"f8e4c1c5aac977d6","entry":"computeIoU","repo":"vision-cair/minigpt-4","repo_kind":"listed","path":"demo_v2.py","file_url":"https://github.com/vision-cair/minigpt-4/blob/HEAD/demo_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f8e4c1c5aac977d6"}},{"code_sha256_prefix":"5c3e6655c3b3e958","entry":"extract_substrings","repo":"vision-cair/minigpt-4","repo_kind":"listed","path":"demo_v2.py","file_url":"https://github.com/vision-cair/minigpt-4/blob/HEAD/demo_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5c3e6655c3b3e958"}},{"code_sha256_prefix":"37438be3369e6493","entry":"is_overlapping","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"37438be3369e6493"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}