{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minigpt4-video-advancing-multimodal-llms-for","title":"MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens","arxiv_id":"2404.03413","date":"2024-04-04","proceeding":null,"authors":["Kirolos Ataallah","Xiaoqian Shen","Eslam Abdelrahman","Essam Sleiman","Deyao Zhu","Jian Ding","Mohamed Elhoseiny"],"abstract":"This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/","url_abs":"https://arxiv.org/abs/2404.03413v1","url_pdf":"https://arxiv.org/pdf/2404.03413v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minigpt4-video-advancing-multimodal-llms-for","repo_url":"https://github.com/Vision-CAIR/MiniGPT4-video","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"minigpt4-video-advancing-multimodal-llms-for","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/2/minigpt4","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"video-based-generative-performance-5","task_name":"Video-based Generative Performance Benchmarking (Consistency)"},{"task_slug":"video-based-generative-performance-3","task_name":"Video-based Generative Performance Benchmarking (Contextual Understanding)"},{"task_slug":"video-based-generative-performance-1","task_name":"Video-based Generative Performance Benchmarking (Correctness of Information)"},{"task_slug":"video-based-generative-performance-2","task_name":"Video-based Generative Performance Benchmarking (Detail Orientation))"},{"task_slug":"video-based-generative-performance-4","task_name":"Video-based Generative Performance Benchmarking (Temporal Understanding)"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"},{"task_slug":"zeroshot-video-question-answer-1","task_name":"Zeroshot Video Question Answer"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-based-generative-performance-2","task":"Video-based Generative Performance Benchmarking (Consistency)","dataset":"VideoInstruct","model":"MiniGPT4-video-7B","rank_in_archive_order":9,"of":18,"metrics":{"gpt-score":"2.67"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-3","task":"Video-based Generative Performance Benchmarking (Contextual Understanding)","dataset":"VideoInstruct","model":"MiniGPT4-video-7B","rank_in_archive_order":8,"of":18,"metrics":{"gpt-score":"3.57"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-1","task":"Video-based Generative Performance Benchmarking (Correctness of Information)","dataset":"VideoInstruct","model":"MiniGPT4-video-7B","rank_in_archive_order":8,"of":18,"metrics":{"gpt-score":"3.08"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-4","task":"Video-based Generative Performance Benchmarking (Detail Orientation))","dataset":"VideoInstruct","model":"MiniGPT4-video-7B","rank_in_archive_order":7,"of":18,"metrics":{"gpt-score":"3.02"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-5","task":"Video-based Generative Performance Benchmarking (Temporal Understanding)","dataset":"VideoInstruct","model":"MiniGPT4-video-7B","rank_in_archive_order":8,"of":18,"metrics":{"gpt-score":"2.65"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-activitynet","task":"Zero-Shot Video Question Answer","dataset":"ActivityNet-QA","model":"MiniGPT4-video-7B","rank_in_archive_order":18,"of":28,"metrics":{"Accuracy":"46.3"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msrvtt-qa","task":"Zero-Shot Video Question Answer","dataset":"MSRVTT-QA","model":"MiniGPT4-video-7B","rank_in_archive_order":14,"of":30,"metrics":{"Accuracy":"59.73"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msvd-qa","task":"Zero-Shot Video Question Answer","dataset":"MSVD-QA","model":"MiniGPT4-video-7B","rank_in_archive_order":13,"of":28,"metrics":{"Accuracy":"73.92"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-tgif-qa","task":"Zero-Shot Video Question Answer","dataset":"TGIF-QA","model":"MiniGPT4-video-7B","rank_in_archive_order":8,"of":14,"metrics":{"Accuracy":"72.22"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-tvqa","task":"Zero-Shot Video Question Answer","dataset":"TVQA","model":"MiniGPT4-video-7B","rank_in_archive_order":3,"of":9,"metrics":{"Accuracy":"54.21"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.03413","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.03413"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-9/tree/main/2/minigpt4","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Vision-CAIR/MiniGPT4-video","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"712ee243951d37df","entry":"format_timestamp","repo":"Vision-CAIR/MiniGPT4-video","repo_kind":"official","path":"minigpt4_video_demo.py","file_url":"https://github.com/Vision-CAIR/MiniGPT4-video/blob/HEAD/minigpt4_video_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"712ee243951d37df"}},{"code_sha256_prefix":"cfa5c09d62fbfd28","entry":"time_to_milliseconds","repo":"Vision-CAIR/MiniGPT4-video","repo_kind":"official","path":"minigpt4_video_demo.py","file_url":"https://github.com/Vision-CAIR/MiniGPT4-video/blob/HEAD/minigpt4_video_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cfa5c09d62fbfd28"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}