{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-estimation-of-the-local-robustness","title":"Characterizing Data Point Vulnerability via Average-Case Robustness","arxiv_id":"2307.13885","date":"2023-07-26","proceeding":null,"authors":["Tessa Han","Suraj Srinivas","Himabindu Lakkaraju"],"abstract":"Studying the robustness of machine learning models is important to ensure consistent model behaviour across real-world settings. To this end, adversarial robustness is a standard framework, which views robustness of predictions through a binary lens: either a worst-case adversarial misclassification exists in the local region around an input, or it does not. However, this binary perspective does not account for the degrees of vulnerability, as data points with a larger number of misclassified examples in their neighborhoods are more vulnerable. In this work, we consider a complementary framework for robustness, called average-case robustness, which measures the fraction of points in a local region that provides consistent predictions. However, computing this quantity is hard, as standard Monte Carlo approaches are inefficient especially for high-dimensional inputs. In this work, we propose the first analytical estimators for average-case robustness for multi-class classifiers. We show empirically that our estimators are accurate and efficient for standard deep learning models and demonstrate their usefulness for identifying vulnerable data points, as well as quantifying robustness bias of models. Overall, our tools provide a complementary view to robustness, improving our ability to characterize model behaviour.","url_abs":"https://arxiv.org/abs/2307.13885v6","url_pdf":"https://arxiv.org/pdf/2307.13885v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-estimation-of-the-local-robustness","repo_url":"https://github.com/ai4life-group/average-case-robustness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"multi-class-classification","task_name":"Multi-class Classification"}],"methods":[{"method_slug":"randomized-smoothing","method_name":"Randomized Smoothing"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.13885","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.13885"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai4life-group/average-case-robustness","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":13,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a3b4819210f94d04","entry":"ResNet18","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3b4819210f94d04"}},{"code_sha256_prefix":"333bf028a3aa2d1f","entry":"ResNet34","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"333bf028a3aa2d1f"}},{"code_sha256_prefix":"c1ab608e5f02e5d3","entry":"ResNet50","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1ab608e5f02e5d3"}},{"code_sha256_prefix":"639a0058eb04fb0d","entry":"add_gaussian_noise","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/fns_add_noise.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/fns_add_noise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"639a0058eb04fb0d"}},{"code_sha256_prefix":"510f78be910a4482","entry":"add_gaussian_noise_batched","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/fns_add_noise.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/fns_add_noise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"510f78be910a4482"}},{"code_sha256_prefix":"b841d740cf91f305","entry":"compute_mvn_cdf","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/fns_mvn_cdf_torch.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/fns_mvn_cdf_torch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b841d740cf91f305"}},{"code_sha256_prefix":"13cc8f2d7e619266","entry":"compute_mvn_cdf_nlp","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/fns_mvn_cdf_torch.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/fns_mvn_cdf_torch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13cc8f2d7e619266"}},{"code_sha256_prefix":"ac24158304cb2789","entry":"f_to_g","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"src/fns_sigma.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/src/fns_sigma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac24158304cb2789"}},{"code_sha256_prefix":"ffa63c6f43ab3b39","entry":"get_list_batch_sizes","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/class_p_emp.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/class_p_emp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffa63c6f43ab3b39"}},{"code_sha256_prefix":"e006e1e840d00026","entry":"gradf_to_gradg","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"estimators/class_p_mmse.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/estimators/class_p_mmse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e006e1e840d00026"}},{"code_sha256_prefix":"d7ee8480d9544bea","entry":"mv_sigmoid","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"src/fns_sigma.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/src/fns_sigma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7ee8480d9544bea"}},{"code_sha256_prefix":"facd8aafc128834a","entry":"round_probust_values","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"calc_p_robust.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/calc_p_robust.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"facd8aafc128834a"}},{"code_sha256_prefix":"18d594adb10a3a11","entry":"run_estimator_on_dataloader","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"calc_p_robust.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/calc_p_robust.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18d594adb10a3a11"}},{"code_sha256_prefix":"705316d160a29f46","entry":"x_to_gradf","repo":"ai4life-group/average-case-robustness","repo_kind":"official","path":"src/fns_sigma.py","file_url":"https://github.com/ai4life-group/average-case-robustness/blob/HEAD/src/fns_sigma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"705316d160a29f46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}