{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lstp-language-guided-spatial-temporal-prompt","title":"Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge","arxiv_id":"2402.16050","date":"2024-02-25","proceeding":null,"authors":["Yuxuan Wang","Yueqian Wang","Pengfei Wu","Jianxin Liang","Dongyan Zhao","Yang Liu","Zilong Zheng"],"abstract":"Despite progress in multimodal large language models (MLLMs), the challenge of interpreting long-form videos in response to linguistic queries persists, largely due to the inefficiency in temporal grounding and limited pre-trained context window size. In this work, we introduce Temporal Grounding Bridge (TGB), a novel framework that bootstraps MLLMs with advanced temporal grounding capabilities and broadens their contextual scope. Our framework significantly enhances the temporal capabilities of current MLLMs through three key innovations: an efficient multi-span temporal grounding algorithm applied to low-dimension temporal features projected from flow; a multimodal length extrapolation training paradigm that utilizes low-dimension temporal features to extend the training context window size; and a bootstrapping framework that bridges our model with pluggable MLLMs without requiring annotation. We validate TGB across seven video benchmarks and demonstrate substantial performance improvements compared with prior MLLMs. Notably, our model, initially trained on sequences of four frames, effectively handles sequences up to 16 longer without sacrificing performance, highlighting its scalability and effectiveness in real-world applications. Our code is publicly available at https://github.com/bigai-nlco/VideoTGB","url_abs":"https://arxiv.org/abs/2402.16050v2","url_pdf":"https://arxiv.org/pdf/2402.16050v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lstp-language-guided-spatial-temporal-prompt","repo_url":"https://github.com/bigai-nlco/lstp-chat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"lstp-language-guided-spatial-temporal-prompt","repo_url":"https://github.com/bigai-nlco/videotgb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[{"method_slug":"sps-1","method_name":"SPS"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"LSTP","rank_in_archive_order":29,"of":47,"metrics":{"Accuracy":"72.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16050","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16050"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigai-nlco/lstp-chat","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigai-nlco/videotgb","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bigai-nlco/VideoTGB","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"eval/inference.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/eval/inference.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"fa6d2859d3461c9e","entry":"get_model_name_from_path","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"demo/utils/model_utils.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/demo/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa6d2859d3461c9e"}},{"code_sha256_prefix":"ebd4b08feabf5d88","entry":"rouge_n","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"src/gadgets/my_metrics.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/src/gadgets/my_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ebd4b08feabf5d88"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"eval/inference.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/eval/inference.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"da69a531c190d5ed","entry":"tokenizer_X_token","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"demo/utils/model_utils.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/demo/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da69a531c190d5ed"}},{"code_sha256_prefix":"6205e41e68ab3848","entry":"crop","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"src/gadgets/functional_video.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/src/gadgets/functional_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6205e41e68ab3848"}},{"code_sha256_prefix":"f1f9c7dfbe0b3105","entry":"resize","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"src/gadgets/functional_video.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/src/gadgets/functional_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f9c7dfbe0b3105"}},{"code_sha256_prefix":"d0e65b9f687e2f97","entry":"resized_crop","repo":"bigai-nlco/VideoTGB","repo_kind":"official","path":"src/gadgets/functional_video.py","file_url":"https://github.com/bigai-nlco/VideoTGB/blob/HEAD/src/gadgets/functional_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0e65b9f687e2f97"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}