{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hidden-biases-of-end-to-end-driving-datasets","title":"Hidden Biases of End-to-End Driving Datasets","arxiv_id":"2412.09602","date":"2024-12-12","proceeding":null,"authors":["Julian Zimmerlin","Jens Beißwenger","Bernhard Jaeger","Andreas Geiger","Kashyap Chitta"],"abstract":"End-to-end driving systems have made rapid progress, but have so far not been applied to the challenging new CARLA Leaderboard 2.0. Further, while there is a large body of literature on end-to-end architectures and training strategies, the impact of the training dataset is often overlooked. In this work, we make a first attempt at end-to-end driving for Leaderboard 2.0. Instead of investigating architectures, we systematically analyze the training dataset, leading to new insights: (1) Expert style significantly affects downstream policy performance. (2) In complex data sets, the frames should not be weighted on the basis of simplistic criteria such as class frequencies. (3) Instead, estimating whether a frame changes the target labels compared to previous frames can reduce the size of the dataset without removing important information. By incorporating these findings, our model ranks first and second respectively on the map and sensors tracks of the 2024 CARLA Challenge, and sets a new state-of-the-art on the Bench2Drive test routes. Finally, we uncover a design flaw in the current evaluation metrics and propose a modification for future challenges. Our dataset, code, and pre-trained models are publicly available at https://github.com/autonomousvision/carla_garage.","url_abs":"https://arxiv.org/abs/2412.09602v2","url_pdf":"https://arxiv.org/pdf/2412.09602v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hidden-biases-of-end-to-end-driving-datasets","repo_url":"https://github.com/autonomousvision/carla_garage","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"bench2drive","task_name":"Bench2Drive"},{"task_slug":"carla-leaderboard-2-0","task_name":"CARLA Leaderboard 2.0"}],"methods":[{"method_slug":"carla","method_name":"CARLA"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.09602","atlas_url":"https://app.syntology.ai/?focus=2412.09602","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.09602"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/autonomousvision/carla_garage","reach":null}],"summary":{"ran_honours":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ea7b7cae43f6a13b","entry":"bicycle_model_forward","repo":"autonomousvision/carla_garage","repo_kind":"official","path":"team_code/sensor_agent.py","file_url":"https://github.com/autonomousvision/carla_garage/blob/HEAD/team_code/sensor_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea7b7cae43f6a13b"}},{"code_sha256_prefix":"ac798cd9ba1d0dfe","entry":"measurement_function_hx","repo":"autonomousvision/carla_garage","repo_kind":"official","path":"team_code/sensor_agent.py","file_url":"https://github.com/autonomousvision/carla_garage/blob/HEAD/team_code/sensor_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac798cd9ba1d0dfe"}},{"code_sha256_prefix":"4d2646540fe5e68a","entry":"state_mean","repo":"autonomousvision/carla_garage","repo_kind":"official","path":"team_code/sensor_agent.py","file_url":"https://github.com/autonomousvision/carla_garage/blob/HEAD/team_code/sensor_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d2646540fe5e68a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}