{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vip3d-end-to-end-visual-trajectory-prediction","title":"ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries","arxiv_id":"2208.01582","date":"2022-08-02","proceeding":"CVPR 2023 1","authors":["Junru Gu","Chenxu Hu","Tianyuan Zhang","Xuanyao Chen","Yilun Wang","Yue Wang","Hang Zhao"],"abstract":"Perception and prediction are two separate modules in the existing autonomous driving systems. They interact with each other via hand-picked features such as agent bounding boxes and trajectories. Due to this separation, prediction, as a downstream module, only receives limited information from the perception module. To make matters worse, errors from the perception modules can propagate and accumulate, adversely affecting the prediction results. In this work, we propose ViP3D, a query-based visual trajectory prediction pipeline that exploits rich information from raw videos to directly predict future trajectories of agents in a scene. ViP3D employs sparse agent queries to detect, track, and predict throughout the pipeline, making it the first fully differentiable vision-based trajectory prediction approach. Instead of using historical feature maps and trajectories, useful information from previous timestamps is encoded in agent queries, which makes ViP3D a concise streaming prediction method. Furthermore, extensive experimental results on the nuScenes dataset show the strong vision-based prediction performance of ViP3D over traditional pipelines and previous end-to-end models.","url_abs":"https://arxiv.org/abs/2208.01582v3","url_pdf":"https://arxiv.org/pdf/2208.01582v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vip3d-end-to-end-visual-trajectory-prediction","repo_url":"https://github.com/Tsinghua-MARS-Lab/ViP3D","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"trajectory-prediction","task_name":"Trajectory Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.01582","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.01582"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Tsinghua-MARS-Lab/ViP3D","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c12fa9dce0d5fd34","entry":"inverse_sigmoid","repo":"Tsinghua-MARS-Lab/ViP3D","repo_kind":"official","path":"plugin/vip3d/models/attention_dert3d.py","file_url":"https://github.com/Tsinghua-MARS-Lab/ViP3D/blob/HEAD/plugin/vip3d/models/attention_dert3d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c12fa9dce0d5fd34"}},{"code_sha256_prefix":"9724d55577f6ca2c","entry":"accuracy","repo":"Tsinghua-MARS-Lab/ViP3D","repo_kind":"official","path":"plugin/vip3d/models/loss.py","file_url":"https://github.com/Tsinghua-MARS-Lab/ViP3D/blob/HEAD/plugin/vip3d/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9724d55577f6ca2c"}},{"code_sha256_prefix":"473fc544db5ddaab","entry":"feature_sampling","repo":"Tsinghua-MARS-Lab/ViP3D","repo_kind":"official","path":"plugin/vip3d/models/attention_dert3d.py","file_url":"https://github.com/Tsinghua-MARS-Lab/ViP3D/blob/HEAD/plugin/vip3d/models/attention_dert3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"473fc544db5ddaab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}