{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mdp3-a-training-free-approach-for-list-wise","title":"MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs","arxiv_id":"2501.02885","date":"2025-01-06","proceeding":null,"authors":["Hui Sun","Shiyin Lu","Huanyu Wang","Qing-Guo Chen","Zhao Xu","Weihua Luo","Kaifu Zhang","Ming Li"],"abstract":"Video large language models (Video-LLMs) have made significant progress in understanding videos. However, processing multiple frames leads to lengthy visual token sequences, presenting challenges such as the limited context length cannot accommodate the entire video, and the inclusion of irrelevant frames hinders visual perception. Hence, effective frame selection is crucial. This paper emphasizes that frame selection should follow three key principles: query relevance, list-wise diversity, and sequentiality. Existing methods, such as uniform frame sampling and query-frame matching, do not capture all of these principles. Thus, we propose Markov decision determinantal point process with dynamic programming (MDP3) for frame selection, a training-free and model-agnostic method that can be seamlessly integrated into existing Video-LLMs. Our method first estimates frame similarities conditioned on the query using a conditional Gaussian kernel within the reproducing kernel Hilbert space~(RKHS). We then apply the determinantal point process~(DPP) to the similarity matrix to capture both query relevance and list-wise diversity. To incorporate sequentiality, we segment the video and apply DPP within each segment, conditioned on the preceding segment selection, modeled as a Markov decision process~(MDP) for allocating selection sizes across segments. Theoretically, MDP3 provides a \\((1 - 1/e)\\)-approximate solution to the NP-hard list-wise frame selection problem with pseudo-polynomial time complexity, demonstrating its efficiency. Empirically, MDP3 significantly outperforms existing methods, verifying its effectiveness and robustness.","url_abs":"https://arxiv.org/abs/2501.02885v1","url_pdf":"https://arxiv.org/pdf/2501.02885v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.02885","atlas_url":"https://app.syntology.ai/?focus=2501.02885","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.02885"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sunh-23/MDP3","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bb0df7539e342928","entry":"MultiGaussianKernel","repo":"sunh-23/MDP3","repo_kind":"found_in_text","path":"vlmeval/smp/mdp3_frame_selector.py","file_url":"https://github.com/sunh-23/MDP3/blob/HEAD/vlmeval/smp/mdp3_frame_selector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb0df7539e342928"}},{"code_sha256_prefix":"df91dcbcd26aafc6","entry":"timer","repo":"sunh-23/MDP3","repo_kind":"found_in_text","path":"vlmeval/smp/mdp3_frame_selector.py","file_url":"https://github.com/sunh-23/MDP3/blob/HEAD/vlmeval/smp/mdp3_frame_selector.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df91dcbcd26aafc6"}},{"code_sha256_prefix":"99fdd30e330f0d07","entry":"MDP3","repo":"sunh-23/MDP3","repo_kind":"found_in_text","path":"vlmeval/smp/mdp3_frame_selector.py","file_url":"https://github.com/sunh-23/MDP3/blob/HEAD/vlmeval/smp/mdp3_frame_selector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99fdd30e330f0d07"}},{"code_sha256_prefix":"b00b3977eab08d7c","entry":"SigLip","repo":"sunh-23/MDP3","repo_kind":"found_in_text","path":"vlmeval/smp/mdp3_frame_selector.py","file_url":"https://github.com/sunh-23/MDP3/blob/HEAD/vlmeval/smp/mdp3_frame_selector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b00b3977eab08d7c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}