{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/federated-continual-instruction-tuning","title":"Federated Continual Instruction Tuning","arxiv_id":"2503.12897","date":"2025-03-17","proceeding":null,"authors":["Haiyang Guo","Fanhu Zeng","Fei Zhu","Wenzhuo LIU","Da-Han Wang","Jian Xu","Xu-Yao Zhang","Cheng-Lin Liu"],"abstract":"A vast amount of instruction tuning data is crucial for the impressive performance of Large Multimodal Models (LMMs), but the associated computational costs and data collection demands during supervised fine-tuning make it impractical for most researchers. Federated learning (FL) has the potential to leverage all distributed data and training resources to reduce the overhead of joint training. However, most existing methods assume a fixed number of tasks, while in real-world scenarios, clients continuously encounter new knowledge and often struggle to retain old tasks due to memory constraints. In this work, we introduce the Federated Continual Instruction Tuning (FCIT) benchmark to model this real-world challenge. Our benchmark includes two realistic scenarios, encompassing four different settings and twelve carefully curated instruction tuning datasets. To address the challenges posed by FCIT, we propose dynamic knowledge organization to effectively integrate updates from different tasks during training and subspace selective activation to allocate task-specific output during inference. Extensive experimental results demonstrate that our proposed method significantly enhances model performance across varying levels of data heterogeneity and catastrophic forgetting. Our source code and dataset will be made publicly available.","url_abs":"https://arxiv.org/abs/2503.12897v1","url_pdf":"https://arxiv.org/pdf/2503.12897v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.12897","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.12897"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Ghy0501/FCIT","reach":null}],"summary":{"ran":7,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e6feb5ec321cc67","entry":"CoINMOEExpert","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e6feb5ec321cc67"}},{"code_sha256_prefix":"b530b6619b1ae8da","entry":"CoINMOELinearA","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b530b6619b1ae8da"}},{"code_sha256_prefix":"e251705baf71307f","entry":"CoINMOELinearB","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e251705baf71307f"}},{"code_sha256_prefix":"e55b3841b7343a24","entry":"Conv2d","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e55b3841b7343a24"}},{"code_sha256_prefix":"67bcd009de9ee47b","entry":"Embedding","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67bcd009de9ee47b"}},{"code_sha256_prefix":"d4633df302edabd4","entry":"Linear","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4633df302edabd4"}},{"code_sha256_prefix":"1bf231da21baa1a7","entry":"LoraLayer","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bf231da21baa1a7"}},{"code_sha256_prefix":"370266fb2cef4f97","entry":"CoINMOELoraLayer","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"370266fb2cef4f97"}},{"code_sha256_prefix":"94aad9dfa1c34393","entry":"CoINMOELoraLinear","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94aad9dfa1c34393"}},{"code_sha256_prefix":"42a4cc04644f2f15","entry":"CoINMOELoraModel","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a4cc04644f2f15"}},{"code_sha256_prefix":"4d037e997f6ce17d","entry":"LoraModel","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d037e997f6ce17d"}},{"code_sha256_prefix":"10b3009f403262c7","entry":"mark_only_lora_as_trainable","repo":"Ghy0501/FCIT","repo_kind":"found_in_text","path":"CoIN/peft/tuners/coinmoelora.py","file_url":"https://github.com/Ghy0501/FCIT/blob/HEAD/CoIN/peft/tuners/coinmoelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10b3009f403262c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}