{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cotracker-it-is-better-to-track-together","title":"CoTracker: It is Better to Track Together","arxiv_id":"2307.07635","date":"2023-07-14","proceeding":null,"authors":["Nikita Karaev","Ignacio Rocco","Benjamin Graham","Natalia Neverova","Andrea Vedaldi","Christian Rupprecht"],"abstract":"We introduce CoTracker, a transformer-based model that tracks a large number of 2D points in long video sequences. Differently from most existing approaches that track points independently, CoTracker tracks them jointly, accounting for their dependencies. We show that joint tracking significantly improves tracking accuracy and robustness, and allows CoTracker to track occluded points and points outside of the camera view. We also introduce several innovations for this class of trackers, including using token proxies that significantly improve memory efficiency and allow CoTracker to track 70k points jointly and simultaneously at inference on a single GPU. CoTracker is an online algorithm that operates causally on short windows. However, it is trained utilizing unrolled windows as a recurrent network, maintaining tracks for long periods of time even when points are occluded or leave the field of view. Quantitatively, CoTracker substantially outperforms prior trackers on standard point-tracking benchmarks.","url_abs":"https://arxiv.org/abs/2307.07635v3","url_pdf":"https://arxiv.org/pdf/2307.07635v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cotracker-it-is-better-to-track-together","repo_url":"https://github.com/facebookresearch/co-tracker","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cotracker-it-is-better-to-track-together","repo_url":"https://github.com/large-trajectory-model/atm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"object-tracking","task_name":"Object Tracking"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"point-tracking","task_name":"Point Tracking"},{"task_slug":"motion-prediction","task_name":"motion prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/point-tracking-on-tap-vid-davis","task":"Point Tracking","dataset":"TAP-Vid-DAVIS","model":"CoTracker","rank_in_archive_order":3,"of":3,"metrics":{"Average Jaccard":"65.9","Average PCK":"79.4","Occlusion Accuracy":"89.9"},"uses_additional_data":false},{"leaderboard":"/sota/point-tracking-on-tap-vid-davis-first","task":"Point Tracking","dataset":"TAP-Vid-DAVIS-First","model":"CoTracker","rank_in_archive_order":2,"of":2,"metrics":{"Average Jaccard":"62.2","Average PCK":"75.7","Occlusion Accuracy":"89.3"},"uses_additional_data":false},{"leaderboard":"/sota/point-tracking-on-tap-vid-kinetics-first","task":"Point Tracking","dataset":"TAP-Vid-Kinetics-First","model":"CoTracker","rank_in_archive_order":2,"of":2,"metrics":{"Average Jaccard":"48.8","Average PCK":"64.5","Occlusion Accuracy":"85.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.07635","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}