{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-video-discovery-agentic-search-with-tool","title":"Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding","arxiv_id":"2505.18079","date":"2025-05-23","proceeding":null,"authors":["Xiaoyi Zhang","Zhaoyang Jia","Zongyu Guo","Jiahao Li","Bin Li","Houqiang Li","Yan Lu"],"abstract":"Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery agent to leverage an agentic search strategy over segmented video clips. Different from previous video agents manually designing a rigid workflow, our approach emphasizes the autonomous nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools, formulates appropriate parameters for actions, and iteratively refines its internal reasoning in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates the advantage of the entire system design. Our DVD agent achieves SOTA performance, significantly surpassing prior works by a large margin on the challenging LVBench dataset. Comprehensive ablation studies and in-depth tool analyses are also provided, yielding insights to further advance intelligent agents tailored for long-form video understanding tasks. The code will be released later.","url_abs":"https://arxiv.org/abs/2505.18079v2","url_pdf":"https://arxiv.org/pdf/2505.18079v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.18079","atlas_url":"https://app.syntology.ai/?focus=2505.18079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.18079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/DeepVideoDiscovery","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10},"by_repo_kind":{"found_in_text":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"27bf334776feb4be","entry":"as_json_schema","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/func_call_shema.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/func_call_shema.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27bf334776feb4be"}},{"code_sha256_prefix":"c907ab3626ac15fd","entry":"assemble_zip","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"reproduce/decode_frames.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/reproduce/decode_frames.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c907ab3626ac15fd"}},{"code_sha256_prefix":"aca54d439555eaaf","entry":"convert_seconds_to_hhmmss","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/frame_caption.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/frame_caption.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aca54d439555eaaf"}},{"code_sha256_prefix":"c7115c8278660151","entry":"doc","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/func_call_shema.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/func_call_shema.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7115c8278660151"}},{"code_sha256_prefix":"3dddbd13049fdd13","entry":"find_all_parts","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"reproduce/decode_frames.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/reproduce/decode_frames.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dddbd13049fdd13"}},{"code_sha256_prefix":"20cc266a8bd12f47","entry":"gather_frames_from_time_ranges","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/frame_caption.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/frame_caption.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20cc266a8bd12f47"}},{"code_sha256_prefix":"87a1b0abc5debf84","entry":"get_parameters_schema","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/func_call_shema.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/func_call_shema.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87a1b0abc5debf84"}},{"code_sha256_prefix":"2d2eb7325321580c","entry":"get_video_id","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"mcp_server.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/mcp_server.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2eb7325321580c"}},{"code_sha256_prefix":"31bb18c44ed2df47","entry":"local_image_to_data_url","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/utils.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31bb18c44ed2df47"}},{"code_sha256_prefix":"d7378512b752bfef","entry":"retry_with_exponential_backoff","repo":"microsoft/DeepVideoDiscovery","repo_kind":"found_in_text","path":"dvd/utils.py","file_url":"https://github.com/microsoft/DeepVideoDiscovery/blob/HEAD/dvd/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7378512b752bfef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}