{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arbitrary-decisions-are-a-hidden-cost-of","title":"Arbitrary Decisions are a Hidden Cost of Differentially Private Training","arxiv_id":"2302.14517","date":"2023-02-28","proceeding":null,"authors":["Bogdan Kulynych","Hsiang Hsu","Carmela Troncoso","Flavio P. Calmon"],"abstract":"Mechanisms used in privacy-preserving machine learning often aim to guarantee differential privacy (DP) during model training. Practical DP-ensuring training methods use randomization when fitting model parameters to privacy-sensitive data (e.g., adding Gaussian noise to clipped gradients). We demonstrate that such randomization incurs predictive multiplicity: for a given input example, the output predicted by equally-private models depends on the randomness used in training. Thus, for a given input, the predicted output can vary drastically if a model is re-trained, even if the same training dataset is used. The predictive-multiplicity cost of DP training has not been studied, and is currently neither audited for nor communicated to model designers and stakeholders. We derive a bound on the number of re-trainings required to estimate predictive multiplicity reliably. We analyze--both theoretically and through extensive experiments--the predictive-multiplicity cost of three DP-ensuring algorithms: output perturbation, objective perturbation, and DP-SGD. We demonstrate that the degree of predictive multiplicity rises as the level of privacy increases, and is unevenly distributed across individuals and demographic groups in the data. Because randomness used to ensure DP during training explains predictions for some examples, our results highlight a fundamental challenge to the justifiability of decisions supported by differentially private models in high-stakes settings. We conclude that practitioners should audit the predictive multiplicity of their DP-ensuring algorithms before deploying them in applications of individual-level consequence.","url_abs":"https://arxiv.org/abs/2302.14517v2","url_pdf":"https://arxiv.org/pdf/2302.14517v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"arbitrary-decisions-are-a-hidden-cost-of","repo_url":"https://github.com/spring-epfl/dp_multiplicity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.14517","atlas_url":"https://app.syntology.ai/?focus=2302.14517","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.14517"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/spring-epfl/dp_multiplicity","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"531dad7054655e86","entry":"binarized_var","repo":"spring-epfl/dp_multiplicity","repo_kind":"official","path":"notebooks/cifar10_dp_analysis.py","file_url":"https://github.com/spring-epfl/dp_multiplicity/blob/HEAD/notebooks/cifar10_dp_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"531dad7054655e86"}},{"code_sha256_prefix":"4540a189fa648127","entry":"class_disc","repo":"spring-epfl/dp_multiplicity","repo_kind":"official","path":"notebooks/cifar10_dp_analysis.py","file_url":"https://github.com/spring-epfl/dp_multiplicity/blob/HEAD/notebooks/cifar10_dp_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4540a189fa648127"}},{"code_sha256_prefix":"d9a731f80983fc21","entry":"process_model","repo":"spring-epfl/dp_multiplicity","repo_kind":"official","path":"notebooks/cifar10_dp_analysis.py","file_url":"https://github.com/spring-epfl/dp_multiplicity/blob/HEAD/notebooks/cifar10_dp_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9a731f80983fc21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}