{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ts-llava-constructing-visual-tokens-through","title":"TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models","arxiv_id":"2411.11066","date":"2024-11-17","proceeding":null,"authors":["Tingyu Qu","Mingxiao Li","Tinne Tuytelaars","Marie-Francine Moens"],"abstract":"Recent advances in multimodal Large Language Models (LLMs) have shown great success in understanding multi-modal contents. For video understanding tasks, training-based video LLMs are difficult to build due to the scarcity of high-quality, curated video-text paired data. In contrast, paired image-text data are much easier to obtain, and there is substantial similarity between images and videos. Consequently, extending image LLMs for video understanding tasks presents an appealing alternative. Developing effective strategies for compressing visual tokens from multiple frames is a promising way to leverage the powerful pre-trained image LLM. In this work, we explore the limitations of the existing compression strategies for building a training-free video LLM. The findings lead to our method TS-LLaVA, which constructs visual tokens through a Thumbnail-and-Sampling strategy. Given a video, we select few equidistant frames from all input frames to construct a Thumbnail image as a detailed visual cue, complemented by Sampled visual tokens from all input frames. Our method establishes the new state-of-the-art performance among training-free video LLMs on various benchmarks. Notably, our 34B model outperforms GPT-4V on the MVBench benchmark, and achieves performance comparable to the 72B training-based video LLM, Video-LLaMA2, on the challenging MLVU benchmark. Code is available at https://github.com/tingyu215/TS-LLaVA.","url_abs":"https://arxiv.org/abs/2411.11066v1","url_pdf":"https://arxiv.org/pdf/2411.11066v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ts-llava-constructing-visual-tokens-through","repo_url":"https://github.com/tingyu215/ts-llava","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"MVBench"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"video-based-generative-performance","task_name":"Video-based Generative Performance Benchmarking"},{"task_slug":"video-based-generative-performance-5","task_name":"Video-based Generative Performance Benchmarking (Consistency)"},{"task_slug":"video-based-generative-performance-3","task_name":"Video-based Generative Performance Benchmarking (Contextual Understanding)"},{"task_slug":"video-based-generative-performance-1","task_name":"Video-based Generative Performance Benchmarking (Correctness of Information)"},{"task_slug":"video-based-generative-performance-2","task_name":"Video-based Generative Performance Benchmarking (Detail Orientation))"},{"task_slug":"video-based-generative-performance-4","task_name":"Video-based Generative Performance Benchmarking (Temporal Understanding)"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-based-generative-performance","task":"Video-based Generative Performance Benchmarking","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":3,"of":23,"metrics":{"mean":"3.38"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-2","task":"Video-based Generative Performance Benchmarking (Consistency)","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":2,"of":18,"metrics":{"gpt-score":"3.69"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-3","task":"Video-based Generative Performance Benchmarking (Contextual Understanding)","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":3,"of":18,"metrics":{"gpt-score":"3.86"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-1","task":"Video-based Generative Performance Benchmarking (Correctness of Information)","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":3,"of":18,"metrics":{"gpt-score":"3.55"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-4","task":"Video-based Generative Performance Benchmarking (Detail Orientation))","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":6,"of":18,"metrics":{"gpt-score":"3.03"},"uses_additional_data":false},{"leaderboard":"/sota/video-based-generative-performance-5","task":"Video-based Generative Performance Benchmarking (Temporal Understanding)","dataset":"VideoInstruct","model":"TS-LLaVA-34B","rank_in_archive_order":5,"of":18,"metrics":{"gpt-score":"2.77"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-activitynet","task":"Zero-Shot Video Question Answer","dataset":"ActivityNet-QA","model":"TS-LLaVA-34B","rank_in_archive_order":6,"of":28,"metrics":{"Accuracy":"58.9","Confidence Score":"3.5"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-egoschema","task":"Zero-Shot Video Question Answer","dataset":"EgoSchema (subset)","model":"TS-LLaVA-34B","rank_in_archive_order":9,"of":14,"metrics":{"Accuracy":"57.8"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-intentqa","task":"Zero-Shot Video Question Answer","dataset":"IntentQA","model":"TS-LLaVA-34B","rank_in_archive_order":3,"of":13,"metrics":{"Accuracy":"67.9"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msrvtt-qa","task":"Zero-Shot Video Question Answer","dataset":"MSRVTT-QA","model":"TS-LLaVA-34B","rank_in_archive_order":7,"of":30,"metrics":{"Accuracy":"66.2","Confidence Score":"3.6"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-msvd-qa","task":"Zero-Shot Video Question Answer","dataset":"MSVD-QA","model":"TS-LLaVA-34B","rank_in_archive_order":8,"of":28,"metrics":{"Accuracy":"79.4","Confidence Score":"4.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-mvbench","task":"Zero-Shot Video Question Answer","dataset":"MVBench","model":"TS-LLaVA-34B","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"52.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-question-answer-on-next-qa","task":"Zero-Shot Video Question Answer","dataset":"NExT-QA","model":"TS-LLaVA-34B","rank_in_archive_order":5,"of":27,"metrics":{"Accuracy":"73.6"},"uses_additional_data":false},{"leaderboard":"/sota/zeroshot-video-question-answer-on-tgif-qa","task":"Zero-Shot Video Question Answer","dataset":"TGIF-QA","model":"TS-LLaVA-34B","rank_in_archive_order":3,"of":14,"metrics":{"Accuracy":"81.0","Confidence Score":"4.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.11066","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.11066"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tingyu215/ts-llava","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":2,"ran_fixture":1,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/eval/run_inference_benchmark_consistency.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/eval/run_inference_benchmark_consistency.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/eval/run_inference_benchmark_consistency.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/eval/run_inference_benchmark_consistency.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"075d76388f7066e2","entry":"find_best_grid","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/model/llava_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"075d76388f7066e2"}},{"code_sha256_prefix":"5764189adab1c8ff","entry":"get_seq_frames","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/eval/model_utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/eval/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5764189adab1c8ff"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"35466d298191b5bd","entry":"resize_images","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/model/llava_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35466d298191b5bd"}},{"code_sha256_prefix":"233b9a2ceb040553","entry":"uniformly_sample_frames","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/model/llava_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"233b9a2ceb040553"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"tingyu215/ts-llava","repo_kind":"official","path":"llava/utils.py","file_url":"https://github.com/tingyu215/ts-llava/blob/HEAD/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}