{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ppllava-varied-video-sequence-understanding","title":"PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance","arxiv_id":"2411.02327","date":"2024-11-04","proceeding":null,"authors":["Ruyang Liu","Haoran Tang","Haibo Liu","Yixiao Ge","Ying Shan","Chen Li","Jiankun Yang"],"abstract":"The past year has witnessed the significant advancement of video-based large language models. However, the challenge of developing a unified model for both short and long video understanding remains unresolved. Most existing video LLMs cannot handle hour-long videos, while methods custom for long videos tend to be ineffective for shorter videos and images. In this paper, we identify the key issue as the redundant content in videos. To address this, we propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three core components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user's instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue. Moreover, our codebase also integrates the most advanced video Direct Preference Optimization (DPO) and visual interleave training. Extensive experiments have validated the performance of our model. With superior throughput and only 1024 visual context, PPLLaVA achieves better results on image benchmarks as a video LLM, while achieving state-of-the-art performance across various video benchmarks, excelling in tasks ranging from caption generation to multiple-choice questions, and handling video lengths from seconds to hours. Codes have been available at https://github.com/farewellthree/PPLLaVA.","url_abs":"https://arxiv.org/abs/2411.02327v2","url_pdf":"https://arxiv.org/pdf/2411.02327v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ppllava-varied-video-sequence-understanding","repo_url":"https://github.com/farewellthree/ppllava","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"video-based-generative-performance","task_name":"Video-based Generative Performance Benchmarking"},{"task_slug":"video-based-generative-performance-5","task_name":"Video-based Generative Performance Benchmarking (Consistency)"},{"task_slug":"video-based-generative-performance-3","task_name":"Video-based Generative Performance Benchmarking (Contextual Understanding)"},{"task_slug":"video-based-generative-performance-1","task_name":"Video-based Generative Performance Benchmarking (Correctness of Information)"},{"task_slug":"video-based-generative-performance-2","task_name":"Video-based Generative Performance Benchmarking (Detail Orientation))"},{"task_slug":"video-based-generative-performance-4","task_name":"Video-based Generative Performance Benchmarking (Temporal Understanding)"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-mvbench","task":"Video Question Answering","dataset":"MVBench","model":"PPLLaVA (7b)","rank_in_archive_order":9,"of":22,"metrics":{"Avg.":"59.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance","task":"Video-based Generative Performance Benchmarking","dataset":"VideoInstruct","model":"PPLLaVA-7B-dpo","rank_in_archive_order":1,"of":23,"metrics":{"Consistency":"3.81","Contextual Understanding":"4.21","Correctness of Information":"3.85","Detail Orientation":"3.56","Temporal Understanding":"3.21","mean":"3.73"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance","task":"Video-based Generative Performance Benchmarking","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":5,"of":23,"metrics":{"Consistency":"3.20","Contextual Understanding":"3.88","Correctness of Information":"3.32","Detail Orientation":"3.20","Temporal Understanding":"3.0","mean":"3.32"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-2","task":"Video-based Generative Performance Benchmarking (Consistency)","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":1,"of":18,"metrics":{"gpt-score":"3.81"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-3","task":"Video-based Generative Performance Benchmarking (Contextual Understanding)","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":1,"of":18,"metrics":{"gpt-score":"4.21"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-1","task":"Video-based Generative Performance Benchmarking (Correctness of Information)","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":1,"of":18,"metrics":{"gpt-score":"3.85"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-4","task":"Video-based Generative Performance Benchmarking (Detail Orientation))","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":1,"of":18,"metrics":{"gpt-score":"3.56"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-5","task":"Video-based Generative Performance Benchmarking (Temporal Understanding)","dataset":"VideoInstruct","model":"PPLLaVA-7B","rank_in_archive_order":1,"of":18,"metrics":{"gpt-score":"3.21"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-activitynet","task":"Zero-Shot Video Question Answer","dataset":"ActivityNet-QA","model":"PPLLaVA-7B","rank_in_archive_order":3,"of":28,"metrics":{"Accuracy":"60.7","Confidence Score":"3.6"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msrvtt-qa","task":"Zero-Shot Video Question Answer","dataset":"MSRVTT-QA","model":"PPLLaVA-7B","rank_in_archive_order":8,"of":30,"metrics":{"Accuracy":"64.3","Confidence Score":"3.5"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msvd-qa","task":"Zero-Shot Video Question Answer","dataset":"MSVD-QA","model":"PPLLaVA-7B","rank_in_archive_order":9,"of":28,"metrics":{"Accuracy":"77.1","Confidence Score":"4.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.02327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/farewellthree/ppllava","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/models/blip2.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/models/base_model.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"e3d9aac39bcef7a7","entry":"create_eva_vit_g","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/models/eva_vit.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/models/eva_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3d9aac39bcef7a7"}},{"code_sha256_prefix":"9a83c0663b6afcd4","entry":"drop_path","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/models/clip_btadapter.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/models/clip_btadapter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a83c0663b6afcd4"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"f4334934ee795269","entry":"is_transformers_greater_than","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/import_utils.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/import_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4334934ee795269"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}},{"code_sha256_prefix":"ded9fb4cd3944865","entry":"tile","repo":"farewellthree/ppllava","repo_kind":"official","path":"ppllava/models/base_model.py","file_url":"https://github.com/farewellthree/ppllava/blob/HEAD/ppllava/models/base_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ded9fb4cd3944865"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}