{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-black-box-assessment-of-autonomous","title":"Efficient Black-box Assessment of Autonomous Vehicle Safety","arxiv_id":"1912.03618","date":"2019-12-08","proceeding":null,"authors":["Justin Norden","Matthew O'Kelly","Aman Sinha"],"abstract":"While autonomous vehicle (AV) technology has shown substantial progress, we still lack tools for rigorous and scalable testing. Real-world testing, the $\\textit{de-facto}$ evaluation method, is dangerous to the public. Moreover, due to the rare nature of failures, billions of miles of driving are needed to statistically validate performance claims. Thus, the industry has largely turned to simulation to evaluate AV systems. However, having a simulation stack alone is not a solution. A simulation testing framework needs to prioritize which scenarios to run, learn how the chosen scenarios provide coverage of failure modes, and rank failure scenarios in order of importance. We implement a simulation testing framework that evaluates an entire modern AV system as a black box. This framework estimates the probability of accidents under a base distribution governing standard traffic behavior. In order to accelerate rare-event probability evaluation, we efficiently learn to identify and rank failure scenarios via adaptive importance-sampling methods. Using this framework, we conduct the first independent evaluation of a full-stack commercial AV system, Comma AI's OpenPilot.","url_abs":"https://arxiv.org/abs/1912.03618v2","url_pdf":"https://arxiv.org/pdf/1912.03618v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-black-box-assessment-of-autonomous","repo_url":"https://github.com/Trustworthy-AI/testpilot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-black-box-assessment-of-autonomous","repo_url":"https://github.com/Trustworthy-AI/trustworthysearch-demo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1912.03618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1912.03618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Trustworthy-AI/trustworthysearch-demo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Trustworthy-AI/testpilot","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"listed":{"samples":12,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d0e191ae9a348b4a","entry":"clip","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/numpy_fast.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/numpy_fast.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0e191ae9a348b4a"}},{"code_sha256_prefix":"b34c1e6652508ae9","entry":"getWorkerIndex","repo":"Trustworthy-AI/trustworthysearch-demo","repo_kind":"listed","path":"python/broker.py","file_url":"https://github.com/Trustworthy-AI/trustworthysearch-demo/blob/HEAD/python/broker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b34c1e6652508ae9"}},{"code_sha256_prefix":"31cbffa1143cf607","entry":"int_rnd","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/numpy_fast.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/numpy_fast.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31cbffa1143cf607"}},{"code_sha256_prefix":"75cbbd8d6697e8ed","entry":"interp","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/numpy_fast.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/numpy_fast.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75cbbd8d6697e8ed"}},{"code_sha256_prefix":"7caa1f53f2d7f7ec","entry":"is_valid_for_fingerprint","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/fingerprints.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/fingerprints.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7caa1f53f2d7f7ec"}},{"code_sha256_prefix":"cc02670c10cca6f7","entry":"json_handler","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/logging_extra.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/logging_extra.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc02670c10cca6f7"}},{"code_sha256_prefix":"3381496eae517481","entry":"json_robust_dumps","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/logging_extra.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/logging_extra.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3381496eae517481"}},{"code_sha256_prefix":"73e73441da88a7b9","entry":"read_db","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/params.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73e73441da88a7b9"}},{"code_sha256_prefix":"c8f3c9a39e388827","entry":"recv_array","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/utils/Utils.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/utils/Utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8f3c9a39e388827"}},{"code_sha256_prefix":"b870fcb7e9e159a7","entry":"send_image","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/utils/Utils.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/utils/Utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b870fcb7e9e159a7"}},{"code_sha256_prefix":"c0a4bd4171aa4a0a","entry":"set_poses","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0a4bd4171aa4a0a"}},{"code_sha256_prefix":"20576537238fa77f","entry":"set_realtime_priority","repo":"Trustworthy-AI/testpilot","repo_kind":"listed","path":"testpilot/testpilot0.5/common/realtime.py","file_url":"https://github.com/Trustworthy-AI/testpilot/blob/HEAD/testpilot/testpilot0.5/common/realtime.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20576537238fa77f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}