{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/futuresightdrive-thinking-visually-with","title":"FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving","arxiv_id":"2505.17685","date":"2025-05-23","proceeding":null,"authors":["Shuang Zeng","Xinyuan Chang","Mengwei Xie","Xinran Liu","Yifan Bai","Zheng Pan","Mu Xu","Xing Wei"],"abstract":"Visual language models (VLMs) have attracted increasing interest in autonomous driving due to their powerful reasoning capabilities. However, existing VLMs typically utilize discrete text Chain-of-Thought (CoT) tailored to the current scenario, which essentially represents highly abstract and symbolic compression of visual information, potentially leading to spatio-temporal relationship ambiguity and fine-grained information loss. Is autonomous driving better modeled on real-world simulation and imagination than on pure symbolic logic? In this paper, we propose a spatio-temporal CoT reasoning method that enables models to think visually. First, VLM serves as a world model to generate unified image frame for predicting future world states: where perception results (e.g., lane divider and 3D detection) represent the future spatial relationships, and ordinary future frame represent the temporal evolution relationships. This spatio-temporal CoT then serves as intermediate reasoning steps, enabling the VLM to function as an inverse dynamics model for trajectory planning based on current observations and future predictions. To implement visual generation in VLMs, we propose a unified pretraining paradigm integrating visual generation and understanding, along with a progressive visual CoT enhancing autoregressive image generation. Extensive experimental results demonstrate the effectiveness of the proposed method, advancing autonomous driving towards visual reasoning.","url_abs":"https://arxiv.org/abs/2505.17685v1","url_pdf":"https://arxiv.org/pdf/2505.17685v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"trajectory-planning","task_name":"Trajectory Planning"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.17685","atlas_url":"https://app.syntology.ai/?focus=2505.17685","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.17685"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MIV-XJTU/FSDrive","reach":null}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"37124eedd502424a","entry":"Downsample","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37124eedd502424a"}},{"code_sha256_prefix":"63c2a9fa53a4e807","entry":"Encoder","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63c2a9fa53a4e807"}},{"code_sha256_prefix":"5b6d9f0747d7c874","entry":"LitEma","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b6d9f0747d7c874"}},{"code_sha256_prefix":"4c7c8720519325ed","entry":"ResnetBlock","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c7c8720519325ed"}},{"code_sha256_prefix":"38a1b1b7d030df65","entry":"Upsample","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38a1b1b7d030df65"}},{"code_sha256_prefix":"56014014c4ed0790","entry":"instantiate_from_config","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56014014c4ed0790"}},{"code_sha256_prefix":"5c22fcb7b2c1bd3b","entry":"AttnBlock","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c22fcb7b2c1bd3b"}},{"code_sha256_prefix":"5f22b6ca52d36c27","entry":"MOVQ","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f22b6ca52d36c27"}},{"code_sha256_prefix":"dc9e571552633f72","entry":"MOVQDecoder","repo":"MIV-XJTU/FSDrive","repo_kind":"found_in_text","path":"MoVQGAN/movqgan/models/vqgan.py","file_url":"https://github.com/MIV-XJTU/FSDrive/blob/HEAD/MoVQGAN/movqgan/models/vqgan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc9e571552633f72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}