{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-asynchronous-distributed-expectation","title":"An Asynchronous Distributed Expectation Maximization Algorithm For Massive Data: The DEM Algorithm","arxiv_id":"1806.07533","date":"2018-06-20","proceeding":null,"authors":["Sanvesh Srivastava","Glen DePalma","Chuanhai Liu"],"abstract":"The family of Expectation-Maximization (EM) algorithms provides a general\napproach to fitting flexible models for large and complex data. The expectation\n(E) step of EM-type algorithms is time-consuming in massive data applications\nbecause it requires multiple passes through the full data. We address this\nproblem by proposing an asynchronous and distributed generalization of the EM\ncalled the Distributed EM (DEM). Using DEM, existing EM-type algorithms are\neasily extended to massive data settings by exploiting the divide-and-conquer\ntechnique and widely available computing power, such as grid computing. The DEM\nalgorithm reserves two groups of computing processes called \\emph{workers} and\n\\emph{managers} for performing the E step and the maximization step (M step),\nrespectively. The samples are randomly partitioned into a large number of\ndisjoint subsets and are stored on the worker processes. The E step of DEM\nalgorithm is performed in parallel on all the workers, and every worker\ncommunicates its results to the managers at the end of local E step. The\nmanagers perform the M step after they have received results from a\n$\\gamma$-fraction of the workers, where $\\gamma$ is a fixed constant in $(0,\n1]$. The sequence of parameter estimates generated by the DEM algorithm retains\nthe attractive properties of EM: convergence of the sequence of parameter\nestimates to a local mode and linear global rate of convergence. Across diverse\nsimulations focused on linear mixed-effects models, the DEM algorithm is\nsignificantly faster than competing EM-type algorithms while having a similar\naccuracy. The DEM algorithm maintains its superior empirical performance on a\nmovie ratings database consisting of 10 million ratings.","url_abs":"http://arxiv.org/abs/1806.07533v1","url_pdf":"http://arxiv.org/pdf/1806.07533v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-asynchronous-distributed-expectation","repo_url":"https://github.com/blayes/DEM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"an-asynchronous-distributed-expectation","repo_url":"https://github.com/2023-MindSpore-1/ms-code-211/tree/main/dem","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"an-asynchronous-distributed-expectation","repo_url":"https://github.com/2023-MindSpore-4/Code3/tree/main/dem","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"an-asynchronous-distributed-expectation","repo_url":"https://github.com/mszpc/dem","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}