{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/partial-variance-reduction-improves-non","title":"On the effectiveness of partial variance reduction in federated learning with heterogeneous data","arxiv_id":"2212.02191","date":"2022-12-05","proceeding":"CVPR 2023 1","authors":["Bo Li","Mikkel N. Schmidt","Tommy S. Alstrøm","Sebastian U. Stich"],"abstract":"Data heterogeneity across clients is a key challenge in federated learning. Prior works address this by either aligning client and server models or using control variates to correct client model drift. Although these methods achieve fast convergence in convex or simple non-convex problems, the performance in over-parameterized models such as deep neural networks is lacking. In this paper, we first revisit the widely used FedAvg algorithm in a deep neural network to understand how data heterogeneity influences the gradient updates across the neural network layers. We observe that while the feature extraction layers are learned efficiently by FedAvg, the substantial diversity of the final classification layers across clients impedes the performance. Motivated by this, we propose to correct model drift by variance reduction only on the final layers. We demonstrate that this significantly outperforms existing benchmarks at a similar or lower communication cost. We furthermore provide proof for the convergence rate of our algorithm.","url_abs":"https://arxiv.org/abs/2212.02191v2","url_pdf":"https://arxiv.org/pdf/2212.02191v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"partial-variance-reduction-improves-non","repo_url":"https://github.com/lyn1874/fedpvr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"partial-variance-reduction-improves-non","repo_url":"https://github.com/lyn1874/fedssyn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.02191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.02191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lyn1874/fedssyn","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lyn1874/fedpvr","reach":null}],"summary":{"ran_draft_wrong":5,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2aa5d536ff654fac","entry":"conv3x3","repo":"lyn1874/fedssyn","repo_kind":"listed","path":"fed_model/resnet.py","file_url":"https://github.com/lyn1874/fedssyn/blob/HEAD/fed_model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa5d536ff654fac"}},{"code_sha256_prefix":"9ef18f8acedec570","entry":"get_dir_name","repo":"lyn1874/fedpvr","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/lyn1874/fedpvr/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ef18f8acedec570"}},{"code_sha256_prefix":"ce23b5f12a1cf119","entry":"get_path_init","repo":"lyn1874/fedpvr","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/lyn1874/fedpvr/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce23b5f12a1cf119"}},{"code_sha256_prefix":"bceaf2c1567c9e01","entry":"get_replace_for_init_path","repo":"lyn1874/fedpvr","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/lyn1874/fedpvr/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bceaf2c1567c9e01"}},{"code_sha256_prefix":"d4682009629cffa4","entry":"norm2d","repo":"lyn1874/fedssyn","repo_kind":"listed","path":"fed_model/resnet.py","file_url":"https://github.com/lyn1874/fedssyn/blob/HEAD/fed_model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4682009629cffa4"}},{"code_sha256_prefix":"1e8fa60fb1e99014","entry":"resnet","repo":"lyn1874/fedssyn","repo_kind":"listed","path":"fed_model/resnet.py","file_url":"https://github.com/lyn1874/fedssyn/blob/HEAD/fed_model/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e8fa60fb1e99014"}},{"code_sha256_prefix":"d046fb57114bfdc7","entry":"vgg","repo":"lyn1874/fedssyn","repo_kind":"listed","path":"fed_model/vgg.py","file_url":"https://github.com/lyn1874/fedssyn/blob/HEAD/fed_model/vgg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d046fb57114bfdc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}