{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/caption-anything-in-video-fine-grained-object","title":"Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting","arxiv_id":"2504.05541","date":"2025-04-07","proceeding":null,"authors":["Yunlong Tang","Jing Bi","Chao Huang","Susan Liang","Daiki Shimada","Hang Hua","Yunzhong Xiao","Yizhi Song","Pinxin Liu","Mingqian Feng","Junjia Guo","Zhuo Liu","Luchuan Song","Ali Vosoughi","Jinxi He","Liu He","Zeliang Zhang","Jiebo Luo","Chenliang Xu"],"abstract":"We present CAT-V (Caption AnyThing in Video), a training-free framework for fine-grained object-centric video captioning that enables detailed descriptions of user-selected objects through time. CAT-V integrates three key components: a Segmenter based on SAMURAI for precise object segmentation across frames, a Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection and temporal analysis, and a Captioner using InternVL-2.5 for generating detailed object-centric descriptions. Through spatiotemporal visual prompts and chain-of-thought reasoning, our framework generates detailed, temporally-aware descriptions of objects' attributes, actions, statuses, interactions, and environmental contexts without requiring additional training data. CAT-V supports flexible user interactions through various visual prompts (points, bounding boxes, and irregular regions) and maintains temporal sensitivity by tracking object states and interactions across different time segments. Our approach addresses limitations of existing video captioning methods, which either produce overly abstract descriptions or lack object-level precision, enabling fine-grained, object-specific descriptions while maintaining temporal coherence and spatial accuracy. The GitHub repository for this project is available at https://github.com/yunlong10/CAT-V","url_abs":"https://arxiv.org/abs/2504.05541v1","url_pdf":"https://arxiv.org/pdf/2504.05541v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"caption-anything-in-video-fine-grained-object","repo_url":"https://github.com/yunlong10/CAT-V","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"boundary-detection","task_name":"Boundary Detection"},{"task_slug":"object","task_name":"Object"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-captioning","task_name":"Video Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.05541","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.05541"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yunlong10/CAT-V","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":13,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7ceaad42da42edd5","entry":"compute_axial_cis","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7ceaad42da42edd5"}},{"code_sha256_prefix":"a6c3974188a5ffb0","entry":"get_1d_sine_pe","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a6c3974188a5ffb0"}},{"code_sha256_prefix":"bb060d29525f90eb","entry":"init_t_xy","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bb060d29525f90eb"}},{"code_sha256_prefix":"f8f09f78b474ee7c","entry":"reshape_for_broadcast","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f8f09f78b474ee7c"}},{"code_sha256_prefix":"4ee20f090e58c425","entry":"select_closest_cond_frames","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4ee20f090e58c425"}},{"code_sha256_prefix":"f9850c1d4acdbb5a","entry":"build_sam2","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f9850c1d4acdbb5a"}},{"code_sha256_prefix":"de8c2a625f263855","entry":"build_sam2_hf","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"de8c2a625f263855"}},{"code_sha256_prefix":"c1b9e144954a57e5","entry":"build_sam2_video_predictor","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c1b9e144954a57e5"}},{"code_sha256_prefix":"f524b5227cf05438","entry":"extract_first_frame","repo":"yunlong10/CAT-V","repo_kind":"official","path":"gradio_app.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/gradio_app.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f524b5227cf05438"}},{"code_sha256_prefix":"f2e72d81f39d65cd","entry":"get_activation_fn","repo":"yunlong10/CAT-V","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f2e72d81f39d65cd"}},{"code_sha256_prefix":"4226ecff44930e51","entry":"get_bounding_box","repo":"yunlong10/CAT-V","repo_kind":"official","path":"gradio_app.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/gradio_app.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4226ecff44930e51"}},{"code_sha256_prefix":"fcdca578d1877a9f","entry":"run_inference_pipeline","repo":"yunlong10/CAT-V","repo_kind":"official","path":"gradio_app.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/gradio_app.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fcdca578d1877a9f"}},{"code_sha256_prefix":"b1177ba14a41c035","entry":"unpack_QAs","repo":"yunlong10/CAT-V","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/yunlong10/CAT-V/blob/HEAD/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b1177ba14a41c035"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}