{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/drivedreamer-towards-real-world-driven-world","title":"DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving","arxiv_id":"2309.09777","date":"2023-09-18","proceeding":null,"authors":["XiaoFeng Wang","Zheng Zhu","Guan Huang","Xinze Chen","Jiagang Zhu","Jiwen Lu"],"abstract":"World models, especially in autonomous driving, are trending and drawing extensive attention due to their capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.","url_abs":"https://arxiv.org/abs/2309.09777v2","url_pdf":"https://arxiv.org/pdf/2309.09777v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"drivedreamer-towards-real-world-driven-world","repo_url":"https://github.com/jeffwang987/drivedreamer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.09777","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.09777"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jeffwang987/drivedreamer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"08eb13f0c4199c14","entry":"custom_collate_fn","repo":"jeffwang987/drivedreamer","repo_kind":"listed","path":"dreamer-train/projects/DriveDreamer/drivedreamer/samplers.py","file_url":"https://github.com/jeffwang987/drivedreamer/blob/HEAD/dreamer-train/projects/DriveDreamer/drivedreamer/samplers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08eb13f0c4199c14"}},{"code_sha256_prefix":"3cc03c7ea3921de9","entry":"get_data","repo":"jeffwang987/drivedreamer","repo_kind":"listed","path":"dreamer-train/projects/DriveDreamer/drivedreamer/tester.py","file_url":"https://github.com/jeffwang987/drivedreamer/blob/HEAD/dreamer-train/projects/DriveDreamer/drivedreamer/tester.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cc03c7ea3921de9"}},{"code_sha256_prefix":"ffdddae9ecf72e03","entry":"get_fourier_embeds_from_boundingbox","repo":"jeffwang987/drivedreamer","repo_kind":"listed","path":"dreamer-models/dreamer_models/models/drivedreamer/position_net.py","file_url":"https://github.com/jeffwang987/drivedreamer/blob/HEAD/dreamer-models/dreamer_models/models/drivedreamer/position_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffdddae9ecf72e03"}},{"code_sha256_prefix":"5b1bd947aa9bfd86","entry":"postprocess_images","repo":"jeffwang987/drivedreamer","repo_kind":"listed","path":"dreamer-train/projects/DriveDreamer/drivedreamer/tester.py","file_url":"https://github.com/jeffwang987/drivedreamer/blob/HEAD/dreamer-train/projects/DriveDreamer/drivedreamer/tester.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1bd947aa9bfd86"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}