{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-and-communication-efficient-federated","title":"Robust and Communication-Efficient Federated Learning from Non-IID Data","arxiv_id":"1903.02891","date":"2019-03-07","proceeding":null,"authors":["Felix Sattler","Simon Wiedemann","Klaus-Robert Müller","Wojciech Samek"],"abstract":"Federated Learning allows multiple parties to jointly train a deep learning\nmodel on their combined data, without any of the participants having to reveal\ntheir local data to a centralized server. This form of privacy-preserving\ncollaborative learning however comes at the cost of a significant communication\noverhead during training. To address this problem, several compression methods\nhave been proposed in the distributed training literature that can reduce the\namount of required communication by up to three orders of magnitude. These\nexisting methods however are only of limited utility in the Federated Learning\nsetting, as they either only compress the upstream communication from the\nclients to the server (leaving the downstream communication uncompressed) or\nonly perform well under idealized conditions such as iid distribution of the\nclient data, which typically can not be found in Federated Learning. In this\nwork, we propose Sparse Ternary Compression (STC), a new compression framework\nthat is specifically designed to meet the requirements of the Federated\nLearning environment. Our experiments on four different learning tasks\ndemonstrate that STC distinctively outperforms Federated Averaging in common\nFederated Learning scenarios where clients either a) hold non-iid data, b) use\nsmall batch sizes during training, or where c) the number of clients is large\nand the participation rate in every communication round is low. We furthermore\nshow that even if the clients hold iid data and use medium sized batches for\ntraining, STC still behaves pareto-superior to Federated Averaging in the sense\nthat it achieves fixed target accuracies on our benchmarks within both fewer\ntraining iterations and a smaller communication budget.","url_abs":"http://arxiv.org/abs/1903.02891v1","url_pdf":"http://arxiv.org/pdf/1903.02891v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-and-communication-efficient-federated","repo_url":"https://github.com/felisat/federated-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.02891","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}