{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/differentially-private-federated-low-rank","title":"Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix","arxiv_id":"2507.09990","date":"2025-07-14","proceeding":null,"authors":["Ming Wen","Jiaqi Zhu","Yuedong Xu","Yipeng Zhou","Dingding Han"],"abstract":"Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers a computationally efficient approach by training low-rank adapters. LoRA is also communication-efficient for federated LLMs when multiple users collaboratively fine-tune a global LLM model without sharing their proprietary raw data. However, even the transmission of local adapters between a server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adapters amplifies synthetic noise on the model, while fixing one adapter impairs the learnability of fine-tuning. In this paper, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double Sketching) , a novel federated LoRA framework to enable effective updating of both low-rank adapters with robust differential privacy. Inspired by randomized SVD, our key idea is a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy-preserving local updates, and then reconstructs the global matrices on the server to facilitate effective updating of both adapters. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions.","url_abs":"https://arxiv.org/abs/2507.09990v1","url_pdf":"https://arxiv.org/pdf/2507.09990v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"differentially-private-federated-low-rank","repo_url":"https://github.com/fleecermw/privacyfedllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2507.09990","atlas_url":"https://app.syntology.ai/?focus=2507.09990","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2507.09990"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/FLEECERmw/PrivacyFedLLM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fleecermw/privacyfedllm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3d09261f535fa80c","entry":"get_Omega","repo":"fleecermw/privacyfedllm","repo_kind":"official","path":"policy/policy_fedask.py","file_url":"https://github.com/fleecermw/privacyfedllm/blob/HEAD/policy/policy_fedask.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d09261f535fa80c"}},{"code_sha256_prefix":"6c317bc0c988fd96","entry":"FedASKPolicy","repo":"fleecermw/privacyfedllm","repo_kind":"official","path":"policy/policy_fedask.py","file_url":"https://github.com/fleecermw/privacyfedllm/blob/HEAD/policy/policy_fedask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c317bc0c988fd96"}},{"code_sha256_prefix":"17c1d63d77f6edc5","entry":"FedPolicy","repo":"fleecermw/privacyfedllm","repo_kind":"official","path":"policy/policy_fedask.py","file_url":"https://github.com/fleecermw/privacyfedllm/blob/HEAD/policy/policy_fedask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17c1d63d77f6edc5"}},{"code_sha256_prefix":"721e1f10a04905ce","entry":"load_config","repo":"FLEECERmw/PrivacyFedLLM","repo_kind":"official","path":"config.py","file_url":"https://github.com/FLEECERmw/PrivacyFedLLM/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"721e1f10a04905ce"}},{"code_sha256_prefix":"afc4247e4dd42a64","entry":"setup_ddp","repo":"FLEECERmw/PrivacyFedLLM","repo_kind":"official","path":"main_sft.py","file_url":"https://github.com/FLEECERmw/PrivacyFedLLM/blob/HEAD/main_sft.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afc4247e4dd42a64"}},{"code_sha256_prefix":"fd9a34654ce589b0","entry":"unwrap_model","repo":"FLEECERmw/PrivacyFedLLM","repo_kind":"official","path":"policy/policy_scaffold.py","file_url":"https://github.com/FLEECERmw/PrivacyFedLLM/blob/HEAD/policy/policy_scaffold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd9a34654ce589b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}