{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-rag-visually-aligned-retrieval","title":"Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension","arxiv_id":"2411.13093","date":"2024-11-20","proceeding":null,"authors":["Yongdong Luo","Xiawu Zheng","Xiao Yang","Guilin Li","Haojia Lin","Jinfa Huang","Jiayi Ji","Fei Chao","Jiebo Luo","Rongrong Ji"],"abstract":"Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.","url_abs":"https://arxiv.org/abs/2411.13093v3","url_pdf":"https://arxiv.org/pdf/2411.13093v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-rag-visually-aligned-retrieval","repo_url":"https://github.com/leon1207/video-rag-master","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"mme","task_name":"MME"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":null,"task_name":"Video MME"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"},{"task_slug":"object-detection-1","task_name":"object-detection"},{"task_slug":null,"task_name":"zero-shot long video breakpoint-mode question answering"},{"task_slug":null,"task_name":"zero-shot long video global-model question answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema-1","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (fullset)","model":"Video-RAG (Based on LLaVA-Video)","rank_in_archive_order":5,"of":29,"metrics":{"Accuracy":"66.7"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-video-mme-1","task":"Zero-Shot Video Question Answer","dataset":"Video-MME","model":"Video-RAG (Based on LLaVA-Video)","rank_in_archive_order":2,"of":11,"metrics":{"Accuracy (%)":"77.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-video-mme","task":"Zero-Shot Video Question Answer","dataset":"Video-MME (w/o subs)","model":"Video-RAG (based on LLaVA-Video)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy (%)":"77.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-zero-shot","task":"Zero-Shot Video Question Answer","dataset":"Zero-shot Video Question Answering on LongVideoBench","model":"Video-RAG (based on LLaVA-Video)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy (% )":"65.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.13093","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.13093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Leon1207/Video-RAG-master","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leon1207/video-rag-master","reach":{"status":"ok"}}],"summary":{"ran_violates":1},"by_repo_kind":{"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"df8ad3ad8671f65e","entry":"get_knowledge_flags","repo":"starsuzi/VideoRAG","repo_kind":"community","path":"generation/inference.py","file_url":"https://github.com/starsuzi/VideoRAG/blob/HEAD/generation/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df8ad3ad8671f65e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}