{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autofednlp-an-efficient-fednlp-framework","title":"FedAdapter: Efficient Federated Learning for Modern NLP","arxiv_id":"2205.10162","date":"2022-05-20","proceeding":null,"authors":["Dongqi Cai","Yaozong Wu","Shangguang Wang","Felix Xiaozhu Lin","Mengwei Xu"],"abstract":"Transformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, our measurements show that FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. A key challenge is to properly configure the depth and width of adapters, to which the training speed and efficiency is highly sensitive. No silver-bullet configuration exists: the optimal choice varies across downstream NLP tasks, desired model accuracy, and mobile resources. To automate adapter configuration, we propose FedAdapter, a framework that enhances the existing FedNLP with two novel designs. First, FedAdapter progressively upgrades the adapter configuration throughout a training session; the principle is to quickly learn shallow knowledge by only training fewer and smaller adapters at the model's top layers, and incrementally learn deep knowledge by incorporating deeper and larger adapters. Second, FedAdapter continuously profiles future adapter configurations by allocating participant devices to trial groups. Extensive experiments show that FedAdapter can reduce FedNLP's model convergence delay to no more than several hours, which is up to 155.5$\\times$ faster compared to vanilla FedNLP and 48$\\times$ faster compared to strong baselines.","url_abs":"https://arxiv.org/abs/2205.10162v2","url_pdf":"https://arxiv.org/pdf/2205.10162v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autofednlp-an-efficient-fednlp-framework","repo_url":"https://github.com/UbiquitousLearning/FedAdapter","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.10162","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.10162"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UbiquitousLearning/FedAdapter","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"ada590a4d77f6b67","entry":"add_args","repo":"UbiquitousLearning/FedAdapter","repo_kind":"official","path":"experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","file_url":"https://github.com/UbiquitousLearning/FedAdapter/blob/HEAD/experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ada590a4d77f6b67"}},{"code_sha256_prefix":"39f533b6d649e5cf","entry":"remove_space","repo":"UbiquitousLearning/FedAdapter","repo_kind":"official","path":"experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","file_url":"https://github.com/UbiquitousLearning/FedAdapter/blob/HEAD/experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39f533b6d649e5cf"}},{"code_sha256_prefix":"39d656b4da1a07ad","entry":"get_acc","repo":"UbiquitousLearning/FedAdapter","repo_kind":"official","path":"experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","file_url":"https://github.com/UbiquitousLearning/FedAdapter/blob/HEAD/experiments/distributed/transformer_exps/run_tc_exps/trial_error.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39d656b4da1a07ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}