{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/one-token-to-seg-them-all-language-instructed","title":"One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos","arxiv_id":"2409.19603","date":"2024-09-29","proceeding":null,"authors":["Zechen Bai","Tong He","Haiyang Mei","Pichao Wang","Ziteng Gao","Joya Chen","Lei Liu","Zheng Zhang","Mike Zheng Shou"],"abstract":"We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed <TRK> token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.","url_abs":"https://arxiv.org/abs/2409.19603v1","url_pdf":"https://arxiv.org/pdf/2409.19603v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"one-token-to-seg-them-all-language-instructed","repo_url":"https://github.com/showlab/videolisa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-tracking","task_name":"Object Tracking"},{"task_slug":"reasoning-segmentation","task_name":"Reasoning Segmentation"},{"task_slug":"referring-video-object-segmentation","task_name":"Referring Video Object Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-object-segmentation","task_name":"Video Object Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-video-object-segmentation-on-long","task":"Referring Video Object Segmentation","dataset":"Long-RVOS","model":"VideoLISA","rank_in_archive_order":7,"of":7,"metrics":{"J&F":"33.1","tIoU":"69.6","vIoU":"28.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.19603","atlas_url":"https://app.syntology.ai/?focus=2409.19603","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.19603"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/showlab/videolisa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":3,"ran":3,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":16,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a1c2083ec2eaeb22","entry":"dice_loss","repo":"showlab/videolisa","repo_kind":"official","path":"model/VideoLISA.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/VideoLISA.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1c2083ec2eaeb22"}},{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"77ff24f1cbcb4f2d","entry":"preprocess","repo":"showlab/videolisa","repo_kind":"official","path":"chat.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/chat.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77ff24f1cbcb4f2d"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"a9292f5d89194794","entry":"sigmoid_ce_loss","repo":"showlab/videolisa","repo_kind":"official","path":"model/VideoLISA.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/VideoLISA.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9292f5d89194794"}},{"code_sha256_prefix":"1b9b4f3219b0ece5","entry":"uniform_sample","repo":"showlab/videolisa","repo_kind":"official","path":"chat.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/chat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b9b4f3219b0ece5"}},{"code_sha256_prefix":"8dc94b9682ebebf9","entry":"build_logger","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8dc94b9682ebebf9"}},{"code_sha256_prefix":"3f50fb8374909f5f","entry":"build_sam_vit_b","repo":"showlab/videolisa","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f50fb8374909f5f"}},{"code_sha256_prefix":"bf61a91555ac5280","entry":"build_sam_vit_h","repo":"showlab/videolisa","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf61a91555ac5280"}},{"code_sha256_prefix":"e2ae4c696b8c968c","entry":"build_sam_vit_l","repo":"showlab/videolisa","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2ae4c696b8c968c"}},{"code_sha256_prefix":"b89c913c2723201f","entry":"parse_args","repo":"showlab/videolisa","repo_kind":"official","path":"merge_lora_weights_and_save_hf_model.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/merge_lora_weights_and_save_hf_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b89c913c2723201f"}},{"code_sha256_prefix":"83529642e413f1c1","entry":"parse_args","repo":"showlab/videolisa","repo_kind":"official","path":"chat.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/chat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83529642e413f1c1"}},{"code_sha256_prefix":"50fab0bafe08b1a9","entry":"parse_args","repo":"showlab/videolisa","repo_kind":"official","path":"train_joint.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/train_joint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50fab0bafe08b1a9"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"showlab/videolisa","repo_kind":"official","path":"model/llava/utils.py","file_url":"https://github.com/showlab/videolisa/blob/HEAD/model/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}