{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vista-a-generalizable-driving-world-model","title":"Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability","arxiv_id":"2405.17398","date":"2024-05-27","proceeding":null,"authors":["Shenyuan Gao","Jiazhi Yang","Li Chen","Kashyap Chitta","Yihang Qiu","Andreas Geiger","Jun Zhang","Hongyang Li"],"abstract":"World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.","url_abs":"https://arxiv.org/abs/2405.17398v5","url_pdf":"https://arxiv.org/pdf/2405.17398v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vista-a-generalizable-driving-world-model","repo_url":"https://github.com/opendrivelab/vista","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vista-a-generalizable-driving-world-model","repo_url":"https://github.com/opendrivelab/uniad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.17398","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17398"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendrivelab/uniad","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendrivelab/vista","reach":null}],"summary":{"ran":6,"ran_draft_wrong":4,"ran_honours":1,"unverified":12},"by_repo_kind":{"official":{"samples":23,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5c64e7e669227c45","entry":"AE3DConv","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c64e7e669227c45"}},{"code_sha256_prefix":"0c3b34cc52a71729","entry":"Conv2DWrapper","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3b34cc52a71729"}},{"code_sha256_prefix":"75aa8f65b96515b2","entry":"LinAttnBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75aa8f65b96515b2"}},{"code_sha256_prefix":"b7508055b9e0c4d0","entry":"LinearAttention","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7508055b9e0c4d0"}},{"code_sha256_prefix":"1eefb4280050c186","entry":"LitEma","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1eefb4280050c186"}},{"code_sha256_prefix":"4c0a080d5b99b21b","entry":"Normalize","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c0a080d5b99b21b"}},{"code_sha256_prefix":"80f84a2f0bef0561","entry":"default","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80f84a2f0bef0561"}},{"code_sha256_prefix":"694551f4a162c92a","entry":"disabled_train","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"694551f4a162c92a"}},{"code_sha256_prefix":"30d11b7b4428ab6e","entry":"get_obj_from_str","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d11b7b4428ab6e"}},{"code_sha256_prefix":"f3ff5ab8d2f07e10","entry":"instantiate_from_config","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3ff5ab8d2f07e10"}},{"code_sha256_prefix":"c91c205a5fcd562b","entry":"partialclass","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c91c205a5fcd562b"}},{"code_sha256_prefix":"65943cc8c6a0822c","entry":"AttnBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65943cc8c6a0822c"}},{"code_sha256_prefix":"77628e616403b945","entry":"Decoder","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77628e616403b945"}},{"code_sha256_prefix":"0827884f986ea385","entry":"DiffusionEngine","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0827884f986ea385"}},{"code_sha256_prefix":"335d93824445f73a","entry":"MemoryEfficientAttnBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"335d93824445f73a"}},{"code_sha256_prefix":"9a1320c38ff3a00b","entry":"MemoryEfficientCrossAttention","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a1320c38ff3a00b"}},{"code_sha256_prefix":"7eeaf0d0942c8b14","entry":"MemoryEfficientCrossAttentionWrapper","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7eeaf0d0942c8b14"}},{"code_sha256_prefix":"30af670fe741ec02","entry":"ResBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30af670fe741ec02"}},{"code_sha256_prefix":"3aba0be80d3762a9","entry":"ResnetBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3aba0be80d3762a9"}},{"code_sha256_prefix":"3996f5eb1a63c585","entry":"TimestepBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3996f5eb1a63c585"}},{"code_sha256_prefix":"634ba52fa75838b3","entry":"VideoDecoder","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"634ba52fa75838b3"}},{"code_sha256_prefix":"d865fe90a754a2c9","entry":"VideoResBlock","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d865fe90a754a2c9"}},{"code_sha256_prefix":"df7cb18ec3f78d30","entry":"make_attn","repo":"opendrivelab/vista","repo_kind":"official","path":"vwm/models/diffusion.py","file_url":"https://github.com/opendrivelab/vista/blob/HEAD/vwm/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df7cb18ec3f78d30"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}