{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lmdrive-closed-loop-end-to-end-driving-with","title":"LMDrive: Closed-Loop End-to-End Driving with Large Language Models","arxiv_id":"2312.07488","date":"2023-12-12","proceeding":"CVPR 2024 1","authors":["Hao Shao","Yuxuan Hu","Letian Wang","Steven L. Waslander","Yu Liu","Hongsheng Li"],"abstract":"Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach \"Artificial General Intelligence\". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive","url_abs":"https://arxiv.org/abs/2312.07488v2","url_pdf":"https://arxiv.org/pdf/2312.07488v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lmdrive-closed-loop-end-to-end-driving-with","repo_url":"https://github.com/opendilab/lmdrive","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"lmdrive-closed-loop-end-to-end-driving-with","repo_url":"https://github.com/deepcs233/visual-cot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[{"slug":"lmdrive","name":"LMDrive","full_name":"LMDrive Dataset"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.07488","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.07488"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepcs233/visual-cot","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/lmdrive","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a8458935002f9648","entry":"add_rect","repo":"opendilab/lmdrive","repo_kind":"official","path":"vision_encoder/render.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/vision_encoder/render.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8458935002f9648"}},{"code_sha256_prefix":"ec66d5c2bef0fece","entry":"convert_grid_to_xy","repo":"opendilab/lmdrive","repo_kind":"official","path":"vision_encoder/render.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/vision_encoder/render.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec66d5c2bef0fece"}},{"code_sha256_prefix":"69493ac7629f05fe","entry":"find_peak_box","repo":"opendilab/lmdrive","repo_kind":"official","path":"vision_encoder/render.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/vision_encoder/render.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69493ac7629f05fe"}},{"code_sha256_prefix":"56f02812d66a0d11","entry":"generate_caption","repo":"opendilab/lmdrive","repo_kind":"official","path":"LAVIS/app/caption.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/LAVIS/app/caption.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56f02812d66a0d11"}},{"code_sha256_prefix":"c882c504d11f2b36","entry":"get_actor_display_name","repo":"opendilab/lmdrive","repo_kind":"official","path":"scenario_runner/manual_control.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/scenario_runner/manual_control.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c882c504d11f2b36"}},{"code_sha256_prefix":"13a43a7d815937e6","entry":"read_img","repo":"opendilab/lmdrive","repo_kind":"official","path":"LAVIS/app/calculate_coco_features.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/LAVIS/app/calculate_coco_features.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13a43a7d815937e6"}},{"code_sha256_prefix":"42b372b019c0d5a0","entry":"retransform","repo":"opendilab/lmdrive","repo_kind":"official","path":"vision_encoder/train_pretrain.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/vision_encoder/train_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42b372b019c0d5a0"}},{"code_sha256_prefix":"413cd3fef973bb3d","entry":"generate_script","repo":"opendilab/lmdrive","repo_kind":"official","path":"data_collection/generate_bashs.py","file_url":"https://github.com/opendilab/lmdrive/blob/HEAD/data_collection/generate_bashs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"413cd3fef973bb3d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}