{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hawkeye-training-video-text-llms-for","title":"HawkEye: Training Video-Text LLMs for Grounding Text in Videos","arxiv_id":"2403.10228","date":"2024-03-15","proceeding":null,"authors":["Yueqian Wang","Xiaojun Meng","Jianxin Liang","Yuxuan Wang","Qun Liu","Dongyan Zhao"],"abstract":"Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.","url_abs":"https://arxiv.org/abs/2403.10228v1","url_pdf":"https://arxiv.org/pdf/2403.10228v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hawkeye-training-video-text-llms-for","repo_url":"https://github.com/yellow-binary-tree/hawkeye","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"video-grounding","task_name":"Video Grounding"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-mvbench","task":"Video Question Answering","dataset":"MVBench","model":"HawkEye","rank_in_archive_order":14,"of":22,"metrics":{"Avg.":"47.55"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.10228","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.10228"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yellow-binary-tree/hawkeye","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_fixture":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7d5f32e88d25e016","entry":"anet_path_mapping","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"configs/dataset_utils.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/configs/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d5f32e88d25e016"}},{"code_sha256_prefix":"01135a3c4116e191","entry":"clevrer_path_mapping","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"configs/dataset_utils.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/configs/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01135a3c4116e191"}},{"code_sha256_prefix":"a475c4129bb51880","entry":"get_sim","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"models/criterions.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/models/criterions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a475c4129bb51880"}},{"code_sha256_prefix":"f64c56c357505158","entry":"interpolate_temporal_pos_embed","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/models/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f64c56c357505158"}},{"code_sha256_prefix":"16812806ff217635","entry":"load_data","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"data_preparing/charades.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/data_preparing/charades.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16812806ff217635"}},{"code_sha256_prefix":"5349fd1148712a53","entry":"load_temp_embed_with_mismatch","repo":"yellow-binary-tree/hawkeye","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/yellow-binary-tree/hawkeye/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5349fd1148712a53"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}