{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/erasurehead-distributed-gradient-descent","title":"ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding","arxiv_id":"1901.09671","date":"2019-01-28","proceeding":null,"authors":["Hongyi Wang","Zachary Charles","Dimitris Papailiopoulos"],"abstract":"We present ErasureHead, a new approach for distributed gradient descent (GD)\nthat mitigates system delays by employing approximate gradient coding. Gradient\ncoded distributed GD uses redundancy to exactly recover the gradient at each\niteration from a subset of compute nodes. ErasureHead instead uses approximate\ngradient codes to recover an inexact gradient at each iteration, but with\nhigher delay tolerance. Unlike prior work on gradient coding, we provide a\nperformance analysis that combines both delay and convergence guarantees. We\nestablish that down to a small noise floor, ErasureHead converges as quickly as\ndistributed GD and has faster overall runtime under a probabilistic delay\nmodel. We conduct extensive experiments on real world datasets and distributed\nclusters and demonstrate that our method can lead to significant speedups over\nboth standard and gradient coded GD.","url_abs":"http://arxiv.org/abs/1901.09671v1","url_pdf":"http://arxiv.org/pdf/1901.09671v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"erasurehead-distributed-gradient-descent","repo_url":"https://github.com/hwang595/ErasureHead","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.09671","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}