{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/private-empirical-risk-minimization-beyond","title":"Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometry","arxiv_id":"1411.5417","date":"2014-11-20","proceeding":null,"authors":["Kunal Talwar","Abhradeep Thakurta","Li Zhang"],"abstract":"Empirical Risk Minimization (ERM) is a standard technique in machine\nlearning, where a model is selected by minimizing a loss function over\nconstraint set. When the training dataset consists of private information, it\nis natural to use a differentially private ERM algorithm, and this problem has\nbeen the subject of a long line of work started with Chaudhuri and Monteleoni\n2008. A private ERM algorithm outputs an approximate minimizer of the loss\nfunction and its error can be measured as the difference from the optimal value\nof the loss function. When the constraint set is arbitrary, the required error\nbounds are fairly well understood \\cite{BassilyST14}. In this work, we show\nthat the geometric properties of the constraint set can be used to derive\nsignificantly better results. Specifically, we show that a differentially\nprivate version of Mirror Descent leads to error bounds of the form\n$\\tilde{O}(G_{\\mathcal{C}}/n)$ for a lipschitz loss function, improving on the\n$\\tilde{O}(\\sqrt{p}/n)$ bounds in Bassily, Smith and Thakurta 2014. Here $p$ is\nthe dimensionality of the problem, $n$ is the number of data points in the\ntraining set, and $G_{\\mathcal{C}}$ denotes the Gaussian width of the\nconstraint set that we optimize over. We show similar improvements for strongly\nconvex functions, and for smooth functions. In addition, we show that when the\nloss function is Lipschitz with respect to the $\\ell_1$ norm and $\\mathcal{C}$\nis $\\ell_1$-bounded, a differentially private version of the Frank-Wolfe\nalgorithm gives error bounds of the form $\\tilde{O}(n^{-2/3})$. This captures\nthe important and common case of sparse linear regression (LASSO), when the\ndata $x_i$ satisfies $|x_i|_{\\infty} \\leq 1$ and we optimize over the $\\ell_1$\nball. We show new lower bounds for this setting, that together with known\nbounds, imply that all our upper bounds are tight.","url_abs":"http://arxiv.org/abs/1411.5417v3","url_pdf":"http://arxiv.org/pdf/1411.5417v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"private-empirical-risk-minimization-beyond","repo_url":"https://github.com/sunblaze-ucb/dpml-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"linear-regression","method_name":"Linear Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1411.5417","atlas_url":"https://app.syntology.ai/?focus=1411.5417","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}