{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2511-04678","title":"Tracking and Understanding Object Transformations","arxiv_id":"2511.04678","date":"2025-11-06","proceeding":"NeurIPS","authors":["Yihong Sun","Xinyu Yang","Jennifer J. Sun","Bharath Hariharan"],"abstract":"Real-world objects frequently undergo state transformations. From an apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects and dynamics. However, existing methods often lose track of the target object after transformation, due to significant changes in object appearance. To address this limitation, we introduce the task of Track Any State: tracking objects through transformations while detecting and describing state changes, accompanied by a new benchmark dataset, VOST-TAS. To tackle this problem, we present TubeletGraph, a zero-shot system that recovers missing objects after transformation and maps out how object states are evolving over time. TubeletGraph first identifies potentially overlooked tracks, and determines whether they should be integrated based on semantic and proximity priors. Then, it reasons about the added tracks and generates a state graph describing each observed transformation. TubeletGraph achieves state-of-the-art tracking performance under transformations, while demonstrating deeper understanding of object transformations and promising capabilities in temporal grounding and semantic reasoning for complex object transformations. Code, additional results, and the benchmark dataset are available at https://tubelet-graph.github.io.","url_abs":"https://arxiv.org/abs/2511.04678","url_pdf":"https://arxiv.org/pdf/2511.04678","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2511.04678","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2511.04678"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/YihongSun/TubeletGraph","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"found_in_text":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d2429dad9985bcd","entry":"demo_run_on_image","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/semantic_sim/compute_sim_fcclip.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/semantic_sim/compute_sim_fcclip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d2429dad9985bcd"}},{"code_sha256_prefix":"be236f2f1ee8d309","entry":"encode_image_from_np","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/vlm/prompt_vlm.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/vlm/prompt_vlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be236f2f1ee8d309"}},{"code_sha256_prefix":"c1bc49141b18d0ca","entry":"generate_rand_colors","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1bc49141b18d0ca"}},{"code_sha256_prefix":"340c8b86253b5706","entry":"get_image_payload","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/vlm/prompt_vlm.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/vlm/prompt_vlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"340c8b86253b5706"}},{"code_sha256_prefix":"ae8c47ab0126353a","entry":"get_text_payload","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/vlm/prompt_vlm.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/vlm/prompt_vlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae8c47ab0126353a"}},{"code_sha256_prefix":"c3316d59d103f658","entry":"make_config","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/get_prediction.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/get_prediction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3316d59d103f658"}},{"code_sha256_prefix":"889ce0fcee2816b2","entry":"make_config","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/tubelet/compute_tubelets_sam.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/tubelet/compute_tubelets_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"889ce0fcee2816b2"}},{"code_sha256_prefix":"019bd91fa532e60a","entry":"run_command","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"quick_run.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/quick_run.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"019bd91fa532e60a"}},{"code_sha256_prefix":"3c3e1d4b4672b2c7","entry":"setup","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/tubelet/compute_tubelets_sam.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/tubelet/compute_tubelets_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c3e1d4b4672b2c7"}},{"code_sha256_prefix":"ee7ac82588ca6632","entry":"setup_cfg","repo":"YihongSun/TubeletGraph","repo_kind":"found_in_text","path":"TubeletGraph/entity_segmentation/cropformer.py","file_url":"https://github.com/YihongSun/TubeletGraph/blob/HEAD/TubeletGraph/entity_segmentation/cropformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee7ac82588ca6632"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}