{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ferret-federated-full-parameter-tuning-at","title":"Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models","arxiv_id":"2409.06277","date":"2024-09-10","proceeding":null,"authors":["Yao Shu","Wenyang Hu","See-Kiong Ng","Bryan Kian Hsiang Low","Fei Richard Yu"],"abstract":"Large Language Models (LLMs) have become indispensable in numerous real-world applications. Unfortunately, fine-tuning these models at scale, especially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing methods often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typically comes at the cost of model accuracy. To address these limitations, we propose federated full-parameter tuning at scale for LLMs (Ferret), the first first-order method with shared randomness to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. Ferret accomplishes this through three aspects: (1) it employs widely applied first-order methods for efficient local updates; (2) it projects these updates into a low-dimensional space to considerably reduce communication overhead; and (3) it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that Ferret significantly enhances the scalability of existing federated full-parameter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accuracy. Our implementation is available at https://github.com/allen4747/Ferret.","url_abs":"https://arxiv.org/abs/2409.06277v2","url_pdf":"https://arxiv.org/pdf/2409.06277v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ferret-federated-full-parameter-tuning-at","repo_url":"https://github.com/allen4747/Ferret","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.06277","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.06277"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/allen4747/Ferret","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"community":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"d515c6ab54f0598c","entry":"are_phrases_similar","repo":"apple/ml-ferret","repo_kind":"community","path":"ferret/eval/eval_flickr_entities.py","file_url":"https://github.com/apple/ml-ferret/blob/HEAD/ferret/eval/eval_flickr_entities.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d515c6ab54f0598c"}},{"code_sha256_prefix":"3b23c33aca6cdead","entry":"decode_bbox_from_caption","repo":"apple/ml-ferret","repo_kind":"community","path":"ferret/eval/eval_flickr_entities.py","file_url":"https://github.com/apple/ml-ferret/blob/HEAD/ferret/eval/eval_flickr_entities.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3b23c33aca6cdead"}},{"code_sha256_prefix":"974d957b09ee69d2","entry":"generate_mask_for_feature","repo":"apple/ml-ferret","repo_kind":"community","path":"ferret/serve/gradio_web_server.py","file_url":"https://github.com/apple/ml-ferret/blob/HEAD/ferret/serve/gradio_web_server.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"974d957b09ee69d2"}},{"code_sha256_prefix":"5a8ea802e3fdbf73","entry":"resize_bbox","repo":"apple/ml-ferret","repo_kind":"community","path":"ferret/eval/eval_flickr_entities.py","file_url":"https://github.com/apple/ml-ferret/blob/HEAD/ferret/eval/eval_flickr_entities.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a8ea802e3fdbf73"}},{"code_sha256_prefix":"32bb456f4cc69e9c","entry":"FerretFramework","repo":"allen4747/Ferret","repo_kind":"official","path":"optimizers/ferret_optimizer.py","file_url":"https://github.com/allen4747/Ferret/blob/HEAD/optimizers/ferret_optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32bb456f4cc69e9c"}},{"code_sha256_prefix":"a449641f4bef63f1","entry":"draw_box","repo":"apple/ml-ferret","repo_kind":"community","path":"ferret/serve/gradio_web_server.py","file_url":"https://github.com/apple/ml-ferret/blob/HEAD/ferret/serve/gradio_web_server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a449641f4bef63f1"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}