Papers › D²: Decentralized Training over Decentralized Data

D²: Decentralized Training over Decentralized Data

1 Jul 2018ICML 2018 7archive 2025-07-28

Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, Ji Liu

While training a machine learning model using multiple workers, each of which collects data from its own data source, it would be useful when the data collected from different workers are unique and different. Ironically, recent analysis of decentralized parallel stochastic gradient descent (D-PSGD) relies on the assumption that the data hosted on different workers are not too different. In this paper, we ask the question: Can we design a decentralized parallel stochastic gradient descent algorithm that is less sensitive to the data variance across workers? In this paper, we present D², a novel decentralized parallel stochastic gradient descent algorithm designed for large data variance \xr{among workers} (imprecisely, “decentralized” data). The core of D² is a variance reduction extension of D-PSGD. It improves the convergence rate from O(σ√(nT) + (nζ²)^(1/3) T^(2/3)) to O(σ√(nT)) where ζ² denotes the variance among data on different workers. As a result, D² is robust to data variance among workers. We empirically evaluated D² on image classification tasks, where each worker has access to only the data of a limited set of labels, and find that D² significantly outperforms D-PSGD.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Results from the paper archive 2025-07-28

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections