{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-surprising-effectiveness-of-multimodal","title":"The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval","arxiv_id":"2406.18113","date":"2024-06-26","proceeding":null,"authors":["Boris Meinardus","Anil Batra","Anna Rohrbach","Marcus Rohrbach"],"abstract":"Recent studies have shown promising results in utilizing multimodal large language models (MLLMs) for computer vision tasks such as object detection and semantic segmentation. However, many challenging video tasks remain under-explored. Video-language tasks necessitate spatial and temporal comprehension and require significant compute. Therefore, prior works have developed complex, highly specialized architectures or leveraged additional input signals such as video transcripts to best encode contextual and temporal information, which limits their generality and can be impractical. One particularly challenging task is video moment retrieval, which requires precise temporal and contextual grounding. This work demonstrates the surprising effectiveness of leveraging image-text pretrained MLLMs for moment retrieval. We introduce Mr. BLIP (Mr. as in Moment Retrieval), a multimodal, single-stage model that requires no expensive video-language pretraining, no additional input signal (e.g., no transcript or audio), and has a simpler and more versatile design than prior state-of-the-art methods. We achieve a new state-of-the-art in moment retrieval on the widely used benchmarks Charades-STA, QVHighlights, and ActivityNet Captions. Notably, we attain over 9% (absolute) higher Recall (at 0.5 and 0.7 IoU) on the challenging long-video multi-moment QVHighlights benchmark. Our code is publicly available.","url_abs":"https://arxiv.org/abs/2406.18113v3","url_pdf":"https://arxiv.org/pdf/2406.18113v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-surprising-effectiveness-of-multimodal","repo_url":"https://github.com/sudo-Boris/mr-Blip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"action-localization","task_name":"Action Localization"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"natural-language-moment-retrieval","task_name":"Natural Language Moment Retrieval"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"blip","method_name":"BLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.18113","atlas_url":"https://app.syntology.ai/?focus=2406.18113","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.18113"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sudo-Boris/mr-Blip","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16b8bc0c4c1c1c87","entry":"compute_gradcam","repo":"sudo-Boris/mr-Blip","repo_kind":"official","path":"lavis/models/blip_models/blip_image_text_matching.py","file_url":"https://github.com/sudo-Boris/mr-Blip/blob/HEAD/lavis/models/blip_models/blip_image_text_matching.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"16b8bc0c4c1c1c87"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}