Papers › CarLLaVA: Vision language models for camera-only closed-loop driving
CarLLaVA: Vision language models for camera-only closed-loop driving
Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, Oleg Sinavski
In this technical report, we present CarLLaVA, a Vision Language Model (VLM) for autonomous driving, developed for the CARLA Autonomous Driving Challenge 2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA architecture as backbone, achieving state-of-the-art closed-loop driving performance with only camera input and without the need for complex or expensive labels. Additionally, we show preliminary results on predicting language commentary alongside the driving output. CarLLaVA uses a semi-disentangled output representation of both path predictions and waypoints, getting the advantages of the path for better lateral control and the waypoints for better longitudinal control. We propose an efficient training recipe to train on large driving datasets without wasting compute on easy, trivial data. CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving Challenge 2.0 outperforming the previous state of the art by 458% and the best concurrent submission by 32.6%.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Bench2Drive | Bench2Drive | SimLingo-Base (CarLLaVa) | Driving Score | 85.94 | #3 of 35 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA | Driving Score | 6.87 | #1 of 4 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA | Infraction Score | 0.42 | #1 of 4 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA | Route Completion | 18.08 | #1 of 4 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA (Map Track) | Driving Score | 6.25 | #2 of 4 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA (Map Track) | Infraction Score | 0.39 | #2 of 4 | Archive leaderboard | report |
| CARLA Leaderboard 2.0 | CARLA | CarLLaVA (Map Track) | Route Completion | 18.89 | #2 of 4 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections