{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/achieving-dimension-free-communication-in","title":"Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization","arxiv_id":"2405.15861","date":"2024-05-24","proceeding":null,"authors":["Zhe Li","Bicheng Ying","Zidong Liu","Chaosheng Dong","Haibo Yang"],"abstract":"Federated Learning (FL) offers a promising framework for collaborative and privacy-preserving machine learning across distributed data sources. However, the substantial communication costs associated with FL significantly challenge its efficiency. Specifically, in each communication round, the communication costs scale linearly with the model's dimension, which presents a formidable obstacle, especially in large model scenarios. Despite various communication-efficient strategies, the intrinsic dimension-dependent communication cost remains a major bottleneck for current FL implementations. This paper proposes a novel dimension-free communication algorithm - DeComFL, which leverages the zeroth-order optimization techniques and reduces the communication cost from $\\mathscr{O}(d)$ to $\\mathscr{O}(1)$ by transmitting only a constant number of scalar values between clients and the server in each round, regardless of the dimension $d$ of the model parameters. Theoretically, in non-convex functions, we prove that our algorithm achieves state-of-the-art rates, which show a linear speedup of the number of clients and local steps under standard assumptions. With additional low effective rank assumption, we can further show the convergence rate is independent of the model dimension $d$ as well. Empirical evaluations, encompassing both classic deep learning training and large language model fine-tuning, demonstrate significant reductions in communication overhead. Notably, DeComFL achieves this by transmitting only around 1MB of data in total between the server and a client to fine-tune a model with billions of parameters. Our code is available at https://github.com/ZidongLiu/DeComFL.","url_abs":"https://arxiv.org/abs/2405.15861v5","url_pdf":"https://arxiv.org/pdf/2405.15861v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"achieving-dimension-free-communication-in","repo_url":"https://github.com/ZidongLiu/DeComFL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/classification-on-boolq","task":"Classification","dataset":"BoolQ","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"62.5%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-boolq","task":"Classification","dataset":"BoolQ","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"61.6%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-rte","task":"Classification","dataset":"RTE","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"60.89%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-rte","task":"Classification","dataset":"RTE","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"57.05%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-sst-2","task":"Classification","dataset":"SST-2","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"90.78%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-sst-2","task":"Classification","dataset":"SST-2","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"85.08%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-wsc","task":"Classification","dataset":"WSC","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"64.16%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-wsc","task":"Classification","dataset":"WSC","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"59.59%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-wic","task":"Classification","dataset":"WiC","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"56.14%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-wic","task":"Classification","dataset":"WiC","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"53.38%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-cb","task":"Classification","dataset":"cb","model":"OPT-1.3B","rank_in_archive_order":1,"of":2,"metrics":{"Test Accuracy":"75.71%"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-cb","task":"Classification","dataset":"cb","model":"OPT-125M","rank_in_archive_order":2,"of":2,"metrics":{"Test Accuracy":"75%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.15861","atlas_url":"https://app.syntology.ai/?focus=2405.15861","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15861"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZidongLiu/DeComFL","reach":null}],"summary":{"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"946f87e367f942e4","entry":"get_warmup_lr","repo":"ZidongLiu/DeComFL","repo_kind":"official","path":"zo_rge_main.py","file_url":"https://github.com/ZidongLiu/DeComFL/blob/HEAD/zo_rge_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"946f87e367f942e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}