{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/group-knowledge-transfer-collaborative","title":"Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge","arxiv_id":"2007.14513","date":"2020-07-28","proceeding":"NeurIPS 2020 12","authors":["Chaoyang He","Murali Annavaram","Salman Avestimehr"],"abstract":"Scaling up the convolutional neural network (CNN) size (e.g., width, depth, etc.) is known to effectively improve model accuracy. However, the large model size impedes training on resource-constrained edge devices. For instance, federated learning (FL) may place undue burden on the compute capability of edge nodes, even though there is a strong practical need for FL due to its privacy and confidentiality properties. To address the resource-constrained reality of edge devices, we reformulate FL as a group knowledge transfer training algorithm, called FedGKT. FedGKT designs a variant of the alternating minimization approach to train small CNNs on edge nodes and periodically transfer their knowledge by knowledge distillation to a large server-side CNN. FedGKT consolidates several advantages into a single framework: reduced demand for edge computation, lower communication bandwidth for large CNNs, and asynchronous training, all while maintaining model accuracy comparable to FedAvg. We train CNNs designed based on ResNet-56 and ResNet-110 using three distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants. Our results show that FedGKT can obtain comparable or even slightly higher accuracy than FedAvg. More importantly, FedGKT makes edge training affordable. Compared to the edge training using FedAvg, FedGKT demands 9 to 17 times less computational power (FLOPs) on edge devices and requires 54 to 105 times fewer parameters in the edge CNN. Our source code is released at FedML (https://fedml.ai).","url_abs":"https://arxiv.org/abs/2007.14513v4","url_pdf":"https://arxiv.org/pdf/2007.14513v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"group-knowledge-transfer-collaborative","repo_url":"https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/distributed/fedgkt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"group-knowledge-transfer-collaborative","repo_url":"https://github.com/vinuni-vishc/feddct","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.14513","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.14513"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/distributed/fedgkt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vinuni-vishc/feddct","reach":null}],"summary":{"ran":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8809d87803cd8180","entry":"KL_Loss","repo":"vinuni-vishc/feddct","repo_kind":"listed","path":"fedml_api/distributed/fedgkt/GKTServerTrainer.py","file_url":"https://github.com/vinuni-vishc/feddct/blob/HEAD/fedml_api/distributed/fedgkt/GKTServerTrainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8809d87803cd8180"}},{"code_sha256_prefix":"e308ac4a9f1743b6","entry":"RunningAverage","repo":"vinuni-vishc/feddct","repo_kind":"listed","path":"fedml_api/distributed/fedgkt/GKTServerTrainer.py","file_url":"https://github.com/vinuni-vishc/feddct/blob/HEAD/fedml_api/distributed/fedgkt/GKTServerTrainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e308ac4a9f1743b6"}},{"code_sha256_prefix":"edeed3edc9c8355b","entry":"accuracy","repo":"vinuni-vishc/feddct","repo_kind":"listed","path":"fedml_api/distributed/fedgkt/GKTServerTrainer.py","file_url":"https://github.com/vinuni-vishc/feddct/blob/HEAD/fedml_api/distributed/fedgkt/GKTServerTrainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edeed3edc9c8355b"}},{"code_sha256_prefix":"2f62c363e1a20bdc","entry":"GKTServerTrainer","repo":"vinuni-vishc/feddct","repo_kind":"listed","path":"fedml_api/distributed/fedgkt/GKTServerTrainer.py","file_url":"https://github.com/vinuni-vishc/feddct/blob/HEAD/fedml_api/distributed/fedgkt/GKTServerTrainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f62c363e1a20bdc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}