{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/internvideo2-scaling-video-foundation-models","title":"InternVideo2: Scaling Foundation Models for Multimodal Video Understanding","arxiv_id":"2403.15377","date":"2024-03-22","proceeding":null,"authors":["Yi Wang","Kunchang Li","Xinhao Li","Jiashuo Yu","Yinan He","Chenting Wang","Guo Chen","Baoqi Pei","Ziang Yan","Rongkun Zheng","Jilan Xu","Zun Wang","Yansong Shi","Tianxiang Jiang","Songze Li","Hongjie Zhang","Yifei HUANG","Yu Qiao","Yali Wang","LiMin Wang"],"abstract":"We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2/.","url_abs":"https://arxiv.org/abs/2403.15377v4","url_pdf":"https://arxiv.org/pdf/2403.15377v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"internvideo2-scaling-video-foundation-models","repo_url":"https://github.com/opengvlab/internvideo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"internvideo2-scaling-video-foundation-models","repo_url":"https://github.com/opengvlab/internvideo2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"},{"task_slug":"video-grounding","task_name":"Video Grounding"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-recognition","task_name":"Video Recognition"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"weakly-supervised-learning","task_name":"Weakly-supervised Learning"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"zero-shot-text-to-audio-retrieval","task_name":"Zero-shot Text to Audio Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"InternVideo2-6B","rank_in_archive_order":3,"of":207,"metrics":{"Acc@1":"92.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"InternVideo2-1B","rank_in_archive_order":4,"of":207,"metrics":{"Acc@1":"91.6"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"InternVideo2-6B","rank_in_archive_order":1,"of":65,"metrics":{"Top-1 Accuracy":"91.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"InternVideo2-1B","rank_in_archive_order":3,"of":65,"metrics":{"Top-1 Accuracy":"91.6"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-700","task":"Action Classification","dataset":"Kinetics-700","model":"InternVideo2-6B","rank_in_archive_order":1,"of":36,"metrics":{"Top-1 Accuracy":"85.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-700","task":"Action Classification","dataset":"Kinetics-700","model":"InternVideo2-1B","rank_in_archive_order":2,"of":36,"metrics":{"Top-1 Accuracy":"85.4"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-mit","task":"Action Classification","dataset":"MIT","model":"InternVideo2-6B","rank_in_archive_order":1,"of":2,"metrics":{"Top 1 Accuracy":"51.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-moments-in-time","task":"Action Classification","dataset":"MiT","model":"InternVideo2-1B","rank_in_archive_order":2,"of":29,"metrics":{"Top 1 Accuracy":"50.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-activitynet","task":"Action Recognition","dataset":"ActivityNet","model":"InternVideo2-6B","rank_in_archive_order":3,"of":16,"metrics":{"mAP":"95.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-hacs","task":"Action Recognition","dataset":"HACS","model":"InternVideo2-6B","rank_in_archive_order":1,"of":8,"metrics":{"Top 1 Accuracy":"97.0"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"InternVideo2-1B","rank_in_archive_order":4,"of":123,"metrics":{"Top-1 Accuracy":"77.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"InternVideo2-6B","rank_in_archive_order":119,"of":123,"metrics":{"GFLOPs":"13321","Parameters":"2131","Top-1 Accuracy":"1","Top-5 Accuracy":"12"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"InternVideo2","rank_in_archive_order":2,"of":29,"metrics":{"Accuracy (5-fold)":"98.6","PRE-TRAINING DATASET":"Multiple","Top-1 Accuracy":"98.6"},"uses_additional_data":true},{"leaderboard":"/sota/moment-retrieval-on-charades-sta","task":"Moment Retrieval","dataset":"Charades-STA","model":"InternVideo2-6B","rank_in_archive_order":5,"of":25,"metrics":{"R@1 IoU=0.5":"70.03","R@1 IoU=0.7":"48.95"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-charades-sta","task":"Moment Retrieval","dataset":"Charades-STA","model":"InternVideo2-1B","rank_in_archive_order":6,"of":25,"metrics":{"R@1 IoU=0.5":"68.36","R@1 IoU=0.7":"45.03"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-qvhighlights","task":"Moment Retrieval","dataset":"QVHighlights","model":"InternVideo2-6B","rank_in_archive_order":5,"of":32,"metrics":{"R@1 IoU=0.5":"71.42","R@1 IoU=0.7":"56.45","mAP":"49.24"},"uses_additional_data":true},{"leaderboard":"/sota/temporal-action-localization-on-activitynet","task":"Temporal Action Localization","dataset":"ActivityNet-1.3","model":"InternVideo2-6B","rank_in_archive_order":5,"of":33,"metrics":{"mAP":"41.2"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-activitynet","task":"Temporal Action Localization","dataset":"ActivityNet-1.3","model":"InternVideo2-1B","rank_in_archive_order":6,"of":33,"metrics":{"mAP":"40.4"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-fineaction","task":"Temporal Action Localization","dataset":"FineAction","model":"InternVideo2-6B","rank_in_archive_order":3,"of":9,"metrics":{"mAP":"27.7"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-hacs","task":"Temporal Action Localization","dataset":"HACS","model":"InternVideo2-6B","rank_in_archive_order":4,"of":12,"metrics":{"Average-mAP":"43.3"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-hacs","task":"Temporal Action Localization","dataset":"HACS","model":"InternVideo2-1B","rank_in_archive_order":6,"of":12,"metrics":{"Average-mAP":"42.4"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-thumos14","task":"Temporal Action Localization","dataset":"THUMOS’14","model":"InternVideo2-6B","rank_in_archive_order":4,"of":42,"metrics":{"Avg mAP (0.3:0.7)":"72.0"},"uses_additional_data":true},{"leaderboard":"/sota/temporal-action-localization-on-thumos14","task":"Temporal Action Localization","dataset":"THUMOS’14","model":"InternVideo2-1B","rank_in_archive_order":7,"of":42,"metrics":{"Avg mAP (0.3:0.7)":"69.8"},"uses_additional_data":true},{"leaderboard":"/sota/text-to-audio-retrieval-on-audiocaps","task":"Text to Audio Retrieval","dataset":"AudioCaps","model":"InternVideo2-6B","rank_in_archive_order":1,"of":11,"metrics":{"R@1":"55.2"},"uses_additional_data":true},{"leaderboard":"/sota/text-to-audio-retrieval-on-clotho","task":"Text to Audio Retrieval","dataset":"Clotho","model":"InternVideo2-6B","rank_in_archive_order":2,"of":12,"metrics":{"R@1":"27.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-grounding-on-qvhighlights","task":"Video Grounding","dataset":"QVHighlights","model":"InternVideo2-6B","rank_in_archive_order":1,"of":7,"metrics":{"R@1,IoU=0.5":"71.42","R@1,IoU=0.7":"56.45"},"uses_additional_data":false},{"leaderboard":"/sota/video-grounding-on-qvhighlights","task":"Video Grounding","dataset":"QVHighlights","model":"InternVideo2-1B","rank_in_archive_order":2,"of":7,"metrics":{"R@1,IoU=0.5":"70.00","R@1,IoU=0.7":"54.45"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-mvbench","task":"Video Question Answering","dataset":"MVBench","model":"InternVideo2","rank_in_archive_order":3,"of":22,"metrics":{"Avg.":"67.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-perception-test","task":"Video Question Answering","dataset":"Perception Test","model":"InternVideo2 (8B)","rank_in_archive_order":3,"of":6,"metrics":{"Accuracy (Top-1)":"63.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"InternVideo2-6B","rank_in_archive_order":1,"of":31,"metrics":{"text-to-video R@1":"74.1","video-to-text R@1":"69.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"InternVideo2-6B","rank_in_archive_order":1,"of":40,"metrics":{"text-to-video R@1":"74.2","video-to-text R@1":"71.9"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"InternVideo2-6B","rank_in_archive_order":1,"of":38,"metrics":{"text-to-video R@1":"46.4","video-to-text R@1":"46.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"InternVideo2-6B","rank_in_archive_order":3,"of":40,"metrics":{"text-to-video R@1":"62.8","video-to-text R@1":"60.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"InternVideo2-6B","rank_in_archive_order":1,"of":24,"metrics":{"text-to-video R@1":"61.4","video-to-text R@1":"85.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-vatex","task":"Video Retrieval","dataset":"VATEX","model":"InternVideo2-6B","rank_in_archive_order":4,"of":13,"metrics":{"text-to-video R@1":"75.5","video-to-text R@1":"89.3"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"InternVideo2-6B","rank_in_archive_order":10,"of":29,"metrics":{"Accuracy":"60.2"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-mvbench","task":"Zero-Shot Video Question Answer","dataset":"MVBench","model":"InternVideo2-1B","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"60.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"InternVideo2-6B","rank_in_archive_order":1,"of":12,"metrics":{"text-to-video R@1":"63.2","text-to-video R@10":"92.5","text-to-video R@5":"85.6","video-to-text R@1":"56.5","video-to-text R@10":"90.3","video-to-text R@5":"82.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"InternVideo2-1B","rank_in_archive_order":2,"of":12,"metrics":{"text-to-video R@1":"60.4","text-to-video R@10":"90.8","text-to-video R@5":"83.9","video-to-text R@1":"54.8","video-to-text R@10":"89.5","video-to-text R@5":"81.5"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"InternVideo2-6B","rank_in_archive_order":1,"of":26,"metrics":{"text-to-video R@1":"57.9","text-to-video R@10":"84.6","text-to-video R@5":"80.0","video-to-text R@1":"57.1","video-to-text R@10":"85.0","video-to-text R@5":"79.9"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"InternVideo2-1B","rank_in_archive_order":2,"of":26,"metrics":{"text-to-video R@1":"57.0","text-to-video R@10":"85.1","text-to-video R@5":"80.0","video-to-text R@1":"54.3","video-to-text R@10":"83.5","video-to-text R@5":"77.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"InternVideo2-6B","rank_in_archive_order":1,"of":16,"metrics":{"text-to-video R@1":"33.8","text-to-video R@10":"62.2","text-to-video R@5":"55.9","video-to-text R@1":"30.1","video-to-text R@10":"54.8","video-to-text R@5":"47.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"InternVideo2-1B","rank_in_archive_order":2,"of":16,"metrics":{"text-to-video R@1":"32.0","text-to-video R@10":"59.4","text-to-video R@5":"52.4","video-to-text R@1":"27.3","video-to-text R@10":"51.6","video-to-text R@5":"44.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"InternVideo2-6B","rank_in_archive_order":1,"of":41,"metrics":{"text-to-video R@1":"55.9","text-to-video R@10":"85.1","text-to-video R@5":"78.3","video-to-text R@1":"53.7","video-to-text R@10":"84.1","video-to-text R@5":"77.5"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"InternVideo2-1B","rank_in_archive_order":3,"of":41,"metrics":{"text-to-video R@1":"51.9","text-to-video R@10":"82.5","text-to-video R@5":"75.3","video-to-text R@1":"50.9","video-to-text R@10":"81.8","video-to-text R@5":"73.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"InternVideo2-6B","rank_in_archive_order":1,"of":14,"metrics":{"text-to-video R@1":"59.3","text-to-video R@10":"89.6","text-to-video R@5":"84.4","video-to-text R@1":"83.1","video-to-text R@10":"97.0","video-to-text R@5":"94.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"InternVideo2-1B","rank_in_archive_order":2,"of":14,"metrics":{"text-to-video R@1":"58.1","text-to-video R@10":"88.4","text-to-video R@5":"83.0","video-to-text R@1":"83.3","video-to-text R@10":"96.9","video-to-text R@5":"94.3"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-vatex","task":"Zero-Shot Video Retrieval","dataset":"VATEX","model":"InternVideo2-6B","rank_in_archive_order":2,"of":5,"metrics":{"text-to-video R@1":"71.5","text-to-video R@10":"97.1","text-to-video R@5":"94.0","video-to-text R@1":"85.3","video-to-text R@10":"99.3","video-to-text R@5":"97.9"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-vatex","task":"Zero-Shot Video Retrieval","dataset":"VATEX","model":"InternVideo2-1B","rank_in_archive_order":3,"of":5,"metrics":{"text-to-video R@1":"70.4","text-to-video R@10":"96.9","text-to-video R@5":"93.4","video-to-text R@1":"85.4","video-to-text R@10":"99.1","video-to-text R@5":"97.6"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.15377","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}