{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simlingo-vision-only-closed-loop-autonomous","title":"SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment","arxiv_id":"2503.09594","date":"2025-03-12","proceeding":"CVPR 2025 1","authors":["Katrin Renz","Long Chen","Elahe Arani","Oleg Sinavski"],"abstract":"Integrating large language models (LLMs) into autonomous driving has attracted significant attention with the hope of improving generalization and explainability. However, existing methods often focus on either driving or vision-language understanding but achieving both high driving performance and extensive language understanding remains challenging. In addition, the dominant approach to tackle vision-language understanding is using visual question answering. However, for autonomous driving, this is only useful if it is aligned with the action space. Otherwise, the model's answers could be inconsistent with its behavior. Therefore, we propose a model that can handle three different tasks: (1) closed-loop driving, (2) vision-language understanding, and (3) language-action alignment. Our model SimLingo is based on a vision language model (VLM) and works using only camera, excluding expensive sensors like LiDAR. SimLingo obtains state-of-the-art performance on the widely used CARLA simulator on the Bench2Drive benchmark and is the winning entry at the CARLA challenge 2024. Additionally, we achieve strong results in a wide variety of language-related tasks while maintaining high driving performance.","url_abs":"https://arxiv.org/abs/2503.09594v1","url_pdf":"https://arxiv.org/pdf/2503.09594v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simlingo-vision-only-closed-loop-autonomous","repo_url":"https://github.com/RenzKa/simlingo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"bench2drive","task_name":"Bench2Drive"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"carla","method_name":"CARLA"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.09594","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.09594"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RenzKa/simlingo","reach":null}],"summary":{"ran":8,"ran_draft_wrong":3,"unverified":3},"by_repo_kind":{"listed":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3cd9d0899118482f","entry":"DrivingAdaptor","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cd9d0899118482f"}},{"code_sha256_prefix":"36d5cb2d42c4019a","entry":"DrivingExample","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36d5cb2d42c4019a"}},{"code_sha256_prefix":"7780149c21221e67","entry":"DrivingInput","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7780149c21221e67"}},{"code_sha256_prefix":"9158c2308af952c2","entry":"DrivingLabel","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9158c2308af952c2"}},{"code_sha256_prefix":"4036898f925c6097","entry":"DrivingOutput","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4036898f925c6097"}},{"code_sha256_prefix":"765126a8ac0e686c","entry":"LanguageAdaptor","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"765126a8ac0e686c"}},{"code_sha256_prefix":"0b408cdf017a0487","entry":"LanguageLabel","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b408cdf017a0487"}},{"code_sha256_prefix":"f949a804aa6563b3","entry":"WaypointInputAdaptor","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f949a804aa6563b3"}},{"code_sha256_prefix":"c8facb81fd5d0bd4","entry":"_gather_from_dict","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8facb81fd5d0bd4"}},{"code_sha256_prefix":"5cc019bdbc5c0e2e","entry":"decode_uint8","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc019bdbc5c0e2e"}},{"code_sha256_prefix":"627f61db7eca94a9","entry":"summarise_losses","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"627f61db7eca94a9"}},{"code_sha256_prefix":"f50fd600a3e066c9","entry":"AdaptorList","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f50fd600a3e066c9"}},{"code_sha256_prefix":"d166584531f5374f","entry":"DrivingModel","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d166584531f5374f"}},{"code_sha256_prefix":"c22b994ccafe6a78","entry":"TrainingOutput","repo":"RenzKa/simlingo","repo_kind":"listed","path":"simlingo_training/models/driving.py","file_url":"https://github.com/RenzKa/simlingo/blob/HEAD/simlingo_training/models/driving.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c22b994ccafe6a78"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}