{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/safely-learning-with-private-data-a-federated","title":"Safely Learning with Private Data: A Federated Learning Framework for Large Language Model","arxiv_id":"2406.14898","date":"2024-06-21","proceeding":null,"authors":["JiaYing Zheng","Hainan Zhang","LingXiang Wang","Wangjie Qiu","Hongwei Zheng","Zhiming Zheng"],"abstract":"Private data, being larger and quality-higher than public data, can greatly improve large language models (LLM). However, due to privacy concerns, this data is often dispersed in multiple silos, making its secure utilization for LLM training a challenge. Federated learning (FL) is an ideal solution for training models with distributed private data, but traditional frameworks like FedAvg are unsuitable for LLM due to their high computational demands on clients. An alternative, split learning, offloads most training parameters to the server while training embedding and output layers locally, making it more suitable for LLM. Nonetheless, it faces significant challenges in security and efficiency. Firstly, the gradients of embeddings are prone to attacks, leading to potential reverse engineering of private data. Furthermore, the server's limitation of handle only one client's training request at a time hinders parallel training, severely impacting training efficiency. In this paper, we propose a Federated Learning framework for LLM, named FL-GLM, which prevents data leakage caused by both server-side and peer-client attacks while improving training efficiency. Specifically, we first place the input block and output block on local client to prevent embedding gradient attacks from server. Secondly, we employ key-encryption during client-server communication to prevent reverse engineering attacks from peer-clients. Lastly, we employ optimization methods like client-batching or server-hierarchical, adopting different acceleration methods based on the actual computational capabilities of the server. Experimental results on NLU and generation tasks demonstrate that FL-GLM achieves comparable metrics to centralized chatGLM model, validating the effectiveness of our federated learning framework.","url_abs":"https://arxiv.org/abs/2406.14898v4","url_pdf":"https://arxiv.org/pdf/2406.14898v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"safely-learning-with-private-data-a-federated","repo_url":"https://github.com/TAP-LLM/SplitFedLLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.14898","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.14898"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/TAP-LLM/SplitFedLLM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mher/flower","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1},"found_in_text":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/client/modeling_llama_A.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/client/modeling_llama_A.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"583539efd6fd01fb","entry":"apply_rotary_pos_emb","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/client/modeling_llama_A.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/client/modeling_llama_A.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"583539efd6fd01fb"}},{"code_sha256_prefix":"48f15e5074b93868","entry":"authenticate","repo":"mher/flower","repo_kind":"found_in_text","path":"flower/utils/authentication.py","file_url":"https://github.com/mher/flower/blob/HEAD/flower/utils/authentication.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"48f15e5074b93868"}},{"code_sha256_prefix":"f070a226c483d2ba","entry":"compute_metrics","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/flclient_llama2.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/flclient_llama2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f070a226c483d2ba"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/client/modeling_llama_A.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/client/modeling_llama_A.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"dbf5a8a81a55f6e9","entry":"validate_auth_option","repo":"mher/flower","repo_kind":"found_in_text","path":"flower/utils/authentication.py","file_url":"https://github.com/mher/flower/blob/HEAD/flower/utils/authentication.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dbf5a8a81a55f6e9"}},{"code_sha256_prefix":"bbadfd71e6cb762a","entry":"get_dataset","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/get_dataloader.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/get_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbadfd71e6cb762a"}},{"code_sha256_prefix":"d3c04ccf9dc3326e","entry":"get_dataset","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/get_dataloader_no_pad.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/get_dataloader_no_pad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3c04ccf9dc3326e"}},{"code_sha256_prefix":"9af9ae92ba15cd8d","entry":"is_flower_envvar","repo":"mher/flower","repo_kind":"found_in_text","path":"flower/command.py","file_url":"https://github.com/mher/flower/blob/HEAD/flower/command.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9af9ae92ba15cd8d"}},{"code_sha256_prefix":"74e0ef4a66fe6e1a","entry":"old_compute_metrics","repo":"TAP-LLM/SplitFedLLM","repo_kind":"official","path":"Fed-Llama-module/flclient_llama2_copy.py","file_url":"https://github.com/TAP-LLM/SplitFedLLM/blob/HEAD/Fed-Llama-module/flclient_llama2_copy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74e0ef4a66fe6e1a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}