{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ml-doctor-holistic-risk-assessment-of","title":"ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models","arxiv_id":"2102.02551","date":"2021-02-04","proceeding":null,"authors":["Yugeng Liu","Rui Wen","Xinlei He","Ahmed Salem","Zhikun Zhang","Michael Backes","Emiliano De Cristofaro","Mario Fritz","Yang Zhang"],"abstract":"Inference attacks against Machine Learning (ML) models allow adversaries to learn sensitive information about training data, model parameters, etc. While researchers have studied, in depth, several kinds of attacks, they have done so in isolation. As a result, we lack a comprehensive picture of the risks caused by the attacks, e.g., the different scenarios they can be applied to, the common factors that influence their performance, the relationship among them, or the effectiveness of possible defenses. In this paper, we fill this gap by presenting a first-of-its-kind holistic risk assessment of different inference attacks against machine learning models. We concentrate on four attacks -- namely, membership inference, model inversion, attribute inference, and model stealing -- and establish a threat model taxonomy. Our extensive experimental evaluation, run on five model architectures and four image datasets, shows that the complexity of the training dataset plays an important role with respect to the attack's performance, while the effectiveness of model stealing and membership inference attacks are negatively correlated. We also show that defenses like DP-SGD and Knowledge Distillation can only mitigate some of the inference attacks. Our analysis relies on a modular re-usable software, ML-Doctor, which enables ML model owners to assess the risks of deploying their models, and equally serves as a benchmark tool for researchers and practitioners.","url_abs":"https://arxiv.org/abs/2102.02551v2","url_pdf":"https://arxiv.org/pdf/2102.02551v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ml-doctor-holistic-risk-assessment-of","repo_url":"https://github.com/liuyugeng/ml-doctor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"inference-attack","task_name":"Inference Attack"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"membership-inference-attack","task_name":"Membership Inference Attack"}],"methods":[{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2102.02551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.02551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuyugeng/ml-doctor","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6504bf3802e2af83","entry":"get_attack_dataset_without_shadow","repo":"liuyugeng/ml-doctor","repo_kind":"official","path":"doctor/meminf.py","file_url":"https://github.com/liuyugeng/ml-doctor/blob/HEAD/doctor/meminf.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6504bf3802e2af83"}},{"code_sha256_prefix":"dbd4383b2b56f823","entry":"train_attack_model","repo":"liuyugeng/ml-doctor","repo_kind":"official","path":"doctor/attrinf.py","file_url":"https://github.com/liuyugeng/ml-doctor/blob/HEAD/doctor/attrinf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbd4383b2b56f823"}},{"code_sha256_prefix":"664dc8c7e8f8f11d","entry":"train_shadow_distillation","repo":"liuyugeng/ml-doctor","repo_kind":"official","path":"doctor/meminf.py","file_url":"https://github.com/liuyugeng/ml-doctor/blob/HEAD/doctor/meminf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"664dc8c7e8f8f11d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}