{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/asynchronous-byzantine-machine-learning-the","title":"Asynchronous Byzantine Machine Learning (the case of SGD)","arxiv_id":"1802.07928","date":"2018-02-22","proceeding":"ICML 2018 7","authors":["Georgios Damaskinos","El Mahdi El Mhamdi","Rachid Guerraoui","Rhicheek Patra","Mahsa Taziki"],"abstract":"Asynchronous distributed machine learning solutions have proven very\neffective so far, but always assuming perfectly functioning workers. In\npractice, some of the workers can however exhibit Byzantine behavior, caused by\nhardware failures, software bugs, corrupt data, or even malicious attacks. We\nintroduce \\emph{Kardam}, the first distributed asynchronous stochastic gradient\ndescent (SGD) algorithm that copes with Byzantine workers. Kardam consists of\ntwo complementary components: a filtering and a dampening component. The first\nis scalar-based and ensures resilience against $\\frac{1}{3}$ Byzantine workers.\nEssentially, this filter leverages the Lipschitzness of cost functions and acts\nas a self-stabilizer against Byzantine workers that would attempt to corrupt\nthe progress of SGD. The dampening component bounds the convergence rate by\nadjusting to stale information through a generic gradient weighting scheme. We\nprove that Kardam guarantees almost sure convergence in the presence of\nasynchrony and Byzantine behavior, and we derive its convergence rate. We\nevaluate Kardam on the CIFAR-100 and EMNIST datasets and measure its overhead\nwith respect to non Byzantine-resilient solutions. We empirically show that\nKardam does not introduce additional noise to the learning procedure but does\ninduce a slowdown (the cost of Byzantine resilience) that we both theoretically\nand empirically show to be less than $f/n$, where $f$ is the number of\nByzantine failures tolerated and $n$ the total number of workers.\nInterestingly, we also empirically observe that the dampening component is\ninteresting in its own right for it enables to build an SGD algorithm that\noutperforms alternative staleness-aware asynchronous competitors in\nenvironments with honest workers.","url_abs":"http://arxiv.org/abs/1802.07928v2","url_pdf":"http://arxiv.org/pdf/1802.07928v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"asynchronous-byzantine-machine-learning-the","repo_url":"https://github.com/LPD-EPFL/kardam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}