{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hierarchical-video-moment-retrieval-and-step","title":"Hierarchical Video-Moment Retrieval and Step-Captioning","arxiv_id":"2303.16406","date":"2023-03-29","proceeding":"CVPR 2023 1","authors":["Abhay Zala","Jaemin Cho","Satwik Kottur","Xilun Chen","Barlas Oğuz","Yasher Mehdad","Mohit Bansal"],"abstract":"There is growing interest in searching for information from large video corpora. Prior works have studied relevant tasks, such as text-based video retrieval, moment retrieval, video summarization, and video captioning in isolation, without an end-to-end setup that can jointly search from video corpora and generate summaries. Such an end-to-end setup would allow for many interesting applications, e.g., a text-based search that finds a relevant video from a video corpus, extracts the most relevant moment from that video, and segments the moment into important steps with captions. To address this, we present the HiREST (HIerarchical REtrieval and STep-captioning) dataset and propose a new benchmark that covers hierarchical information retrieval and visual/textual stepwise summarization from an instructional video corpus. HiREST consists of 3.4K text-video pairs from an instructional video dataset, where 1.1K videos have annotations of moment spans relevant to text query and breakdown of each moment into key instruction steps with caption and timestamps (totaling 8.6K step captions). Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks. In moment segmentation, models break down a video moment into instruction steps and identify start-end boundaries. In step captioning, models generate a textual summary for each step. We also present starting point task-specific and end-to-end joint baseline models for our new benchmark. While the baseline models show some promising results, there still exists large room for future improvement by the community. Project website: https://hirest-cvpr2023.github.io","url_abs":"https://arxiv.org/abs/2303.16406v1","url_pdf":"https://arxiv.org/pdf/2303.16406v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hierarchical-video-moment-retrieval-and-step","repo_url":"https://github.com/j-min/HiREST","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-summarization","task_name":"Video Summarization"}],"methods":[],"datasets_introduced":[{"slug":"hirest","name":"HiREST","full_name":"HIerarchical REtrieval and STep-captioning"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.16406","atlas_url":"https://app.syntology.ai/?focus=2303.16406","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16406"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/j-min/HiREST","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41c46b0d5713eb14","entry":"all_gather","repo":"j-min/HiREST","repo_kind":"official","path":"dist_utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41c46b0d5713eb14"}},{"code_sha256_prefix":"e84b722650b81091","entry":"compute_iou","repo":"j-min/HiREST","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e84b722650b81091"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"j-min/HiREST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"2fcba9a712be1836","entry":"frame_index_to_timestamp","repo":"j-min/HiREST","repo_kind":"official","path":"hirest_dataset.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/hirest_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fcba9a712be1836"}},{"code_sha256_prefix":"00a9f7918ac39329","entry":"gather","repo":"j-min/HiREST","repo_kind":"official","path":"dist_utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00a9f7918ac39329"}},{"code_sha256_prefix":"db63032fdd04245b","entry":"get_area","repo":"j-min/HiREST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db63032fdd04245b"}},{"code_sha256_prefix":"924d4471d950b88f","entry":"get_relative_distance","repo":"j-min/HiREST","repo_kind":"official","path":"utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"924d4471d950b88f"}},{"code_sha256_prefix":"1e89d4c87725a5eb","entry":"load_data","repo":"j-min/HiREST","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e89d4c87725a5eb"}},{"code_sha256_prefix":"44f4152cc7eb1ddd","entry":"reduce_dict","repo":"j-min/HiREST","repo_kind":"official","path":"dist_utils.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44f4152cc7eb1ddd"}},{"code_sha256_prefix":"8a0d3c4ffde35ee2","entry":"timestamp_to_frame_index","repo":"j-min/HiREST","repo_kind":"official","path":"hirest_dataset.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/hirest_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a0d3c4ffde35ee2"}},{"code_sha256_prefix":"88e56e4d0c6044f2","entry":"build_model","repo":"j-min/HiREST","repo_kind":"official","path":"EVA_clip/model.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/EVA_clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88e56e4d0c6044f2"}},{"code_sha256_prefix":"45808df80214028e","entry":"evaluate_video_retrieval","repo":"j-min/HiREST","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/j-min/HiREST/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45808df80214028e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}