{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/editable-scene-simulation-for-autonomous","title":"Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents","arxiv_id":"2402.05746","date":"2024-02-08","proceeding":"CVPR 2024 1","authors":["Yuxi Wei","Zi Wang","Yifan Lu","Chenxin Xu","Changxing Liu","Hao Zhao","Siheng Chen","Yanfeng Wang"],"abstract":"Scene simulation in autonomous driving has gained significant attention because of its huge potential for generating customized data. However, existing editable scene simulation approaches face limitations in terms of user interaction efficiency, multi-camera photo-realistic rendering and external digital assets integration. To address these challenges, this paper introduces ChatSim, the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. To enable editing with high command flexibility,~ChatSim leverages a large language model (LLM) agent collaboration framework. To generate photo-realistic outcomes, ChatSim employs a novel multi-camera neural radiance field method. Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering. Our experiments on Waymo Open Dataset demonstrate that ChatSim can handle complex language commands and generate corresponding photo-realistic scene videos.","url_abs":"https://arxiv.org/abs/2402.05746v3","url_pdf":"https://arxiv.org/pdf/2402.05746v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"editable-scene-simulation-for-autonomous","repo_url":"https://github.com/yifanlu0227/chatsim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"lighting-estimation","task_name":"Lighting Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05746","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05746"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yifanlu0227/chatsim","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yifanlu0227/ChatSim","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"2f7a45ac7953e329","entry":"read_yaml","repo":"yifanlu0227/ChatSim","repo_kind":"official","path":"chatsim/agents/utils.py","file_url":"https://github.com/yifanlu0227/ChatSim/blob/HEAD/chatsim/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f7a45ac7953e329"}},{"code_sha256_prefix":"24378caad2616f79","entry":"rotate","repo":"yifanlu0227/ChatSim","repo_kind":"official","path":"chatsim/agents/utils.py","file_url":"https://github.com/yifanlu0227/ChatSim/blob/HEAD/chatsim/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24378caad2616f79"}},{"code_sha256_prefix":"61e40523b4bdb1f6","entry":"transform_nerf2opencv_convention","repo":"yifanlu0227/ChatSim","repo_kind":"official","path":"chatsim/agents/utils.py","file_url":"https://github.com/yifanlu0227/ChatSim/blob/HEAD/chatsim/agents/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61e40523b4bdb1f6"}},{"code_sha256_prefix":"659bc4e7e63ed8f9","entry":"rotmat2qvec","repo":"yifanlu0227/ChatSim","repo_kind":"official","path":"data_utils/transform2colmap_chatsim.py","file_url":"https://github.com/yifanlu0227/ChatSim/blob/HEAD/data_utils/transform2colmap_chatsim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"659bc4e7e63ed8f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}