{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fedbiot-llm-local-fine-tuning-in-federated","title":"FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model","arxiv_id":"2406.17706","date":"2024-06-25","proceeding":null,"authors":["Feijie Wu","Zitao Li","Yaliang Li","Bolin Ding","Jing Gao"],"abstract":"Large language models (LLMs) show amazing performance on many domain-specific tasks after fine-tuning with some appropriate data. However, many domain-specific data are privately distributed across multiple owners. Thus, this dilemma raises the interest in how to perform LLM fine-tuning in federated learning (FL). However, confronted with limited computation and communication capacities, FL clients struggle to fine-tune an LLM effectively. To this end, we introduce FedBiOT, a resource-efficient LLM fine-tuning approach to FL. Specifically, our method involves the server generating a compressed LLM and aligning its performance with the full model. Subsequently, the clients fine-tune a lightweight yet important part of the compressed model, referred to as an adapter. Notice that as the server has no access to the private data owned by the clients, the data used for alignment by the server has a different distribution from the one used for fine-tuning by clients. We formulate the problem into a bi-level optimization problem to minimize the negative effect of data discrepancy and derive the updating rules for the server and clients. We conduct extensive experiments on LLaMA-2, empirically showing that the adapter has exceptional performance when reintegrated into the global LLM. The results also indicate that the proposed FedBiOT significantly reduces resource consumption compared to existing benchmarks, all while achieving comparable performance levels.","url_abs":"https://arxiv.org/abs/2406.17706v1","url_pdf":"https://arxiv.org/pdf/2406.17706v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fedbiot-llm-local-fine-tuning-in-federated","repo_url":"https://github.com/HarliWu/FedBiOT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.17706","atlas_url":"https://app.syntology.ai/?focus=2406.17706","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.17706"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HarliWu/FedBiOT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"93bfa4c750a44d4d","entry":"parse_client_cfg","repo":"HarliWu/FedBiOT","repo_kind":"official","path":"federatedscope/core/cmd_args.py","file_url":"https://github.com/HarliWu/FedBiOT/blob/HEAD/federatedscope/core/cmd_args.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93bfa4c750a44d4d"}},{"code_sha256_prefix":"385ae669172a8fe2","entry":"wrap_regularized_optimizer","repo":"HarliWu/FedBiOT","repo_kind":"official","path":"federatedscope/core/optimizer.py","file_url":"https://github.com/HarliWu/FedBiOT/blob/HEAD/federatedscope/core/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"385ae669172a8fe2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}