{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distributed-statistical-machine-learning-in","title":"Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent","arxiv_id":"1705.05491","date":"2017-05-16","proceeding":null,"authors":["Yudong Chen","Lili Su","Jiaming Xu"],"abstract":"We consider the problem of distributed statistical machine learning in\nadversarial settings, where some unknown and time-varying subset of working\nmachines may be compromised and behave arbitrarily to prevent an accurate model\nfrom being learned. This setting captures the potential adversarial attacks\nfaced by Federated Learning -- a modern machine learning paradigm that is\nproposed by Google researchers and has been intensively studied for ensuring\nuser privacy. Formally, we focus on a distributed system consisting of a\nparameter server and $m$ working machines. Each working machine keeps $N/m$\ndata samples, where $N$ is the total number of samples. The goal is to\ncollectively learn the underlying true model parameter of dimension $d$.\n  In classical batch gradient descent methods, the gradients reported to the\nserver by the working machines are aggregated via simple averaging, which is\nvulnerable to a single Byzantine failure. In this paper, we propose a Byzantine\ngradient descent method based on the geometric median of means of the\ngradients. We show that our method can tolerate $q \\le (m-1)/2$ Byzantine\nfailures, and the parameter estimate converges in $O(\\log N)$ rounds with an\nestimation error of $\\sqrt{d(2q+1)/N}$, hence approaching the optimal error\nrate $\\sqrt{d/N}$ in the centralized and failure-free setting. The total\ncomputational complexity of our algorithm is of $O((Nd/m) \\log N)$ at each\nworking machine and $O(md + kd \\log^3 N)$ at the central server, and the total\ncommunication cost is of $O(m d \\log N)$. We further provide an application of\nour general results to the linear regression problem.\n  A key challenge arises in the above problem is that Byzantine failures create\narbitrary and unspecified dependency among the iterations and the aggregated\ngradients. We prove that the aggregated gradient converges uniformly to the\ntrue gradient function.","url_abs":"http://arxiv.org/abs/1705.05491v2","url_pdf":"http://arxiv.org/pdf/1705.05491v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distributed-statistical-machine-learning-in","repo_url":"https://github.com/bladesteam/blades","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"distributed-statistical-machine-learning-in","repo_url":"https://github.com/lishenghui/blades","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[{"method_slug":"linear-regression","method_name":"Linear Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1705.05491","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}