{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/feddat-an-approach-for-foundation-model","title":"FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning","arxiv_id":"2308.12305","date":"2023-08-21","proceeding":null,"authors":["Haokun Chen","Yao Zhang","Denis Krompass","Jindong Gu","Volker Tresp"],"abstract":"Recently, foundation models have exhibited remarkable advancements in multi-modal learning. These models, equipped with millions (or billions) of parameters, typically require a substantial amount of data for finetuning. However, collecting and centralizing training data from diverse sectors becomes challenging due to distinct privacy regulations. Federated Learning (FL) emerges as a promising solution, enabling multiple clients to collaboratively train neural networks without centralizing their local data. To alleviate client computation burdens and communication overheads, previous works have adapted Parameter-efficient Finetuning (PEFT) methods for FL. Hereby, only a small fraction of the model parameters are optimized and communicated during federated communications. Nevertheless, most previous works have focused on a single modality and neglected one common phenomenon, i.e., the presence of data heterogeneity across the clients. Therefore, in this work, we propose a finetuning framework tailored to heterogeneous multi-modal FL, called Federated Dual-Aadapter Teacher (FedDAT). Specifically, our approach leverages a Dual-Adapter Teacher (DAT) to address data heterogeneity by regularizing the client local updates and applying Mutual Knowledge Distillation (MKD) for an efficient knowledge transfer. FedDAT is the first approach that enables an efficient distributed finetuning of foundation models for a variety of heterogeneous Vision-Language tasks. To demonstrate its effectiveness, we conduct extensive experiments on four multi-modality FL benchmarks with different types of data heterogeneity, where FedDAT substantially outperforms the existing centralized PEFT methods adapted for FL.","url_abs":"https://arxiv.org/abs/2308.12305v1","url_pdf":"https://arxiv.org/pdf/2308.12305v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"feddat-an-approach-for-foundation-model","repo_url":"https://github.com/HaokunChen245/FedDAT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.12305","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.12305"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HaokunChen245/FedDAT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a03026e2eef29f0c","entry":"convert_batch_to_albef_input_dict","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/albef.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/albef.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a03026e2eef29f0c"}},{"code_sha256_prefix":"1bc059a81f1b025b","entry":"convert_batch_to_vilt_input_dict","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/vilt.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/vilt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc059a81f1b025b"}},{"code_sha256_prefix":"9196a812096e2d7a","entry":"convert_batch_to_viltbert_input_dict","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/viltbert.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/viltbert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9196a812096e2d7a"}},{"code_sha256_prefix":"b55bd7c166344bfe","entry":"BERTEmbeddings_prompted_forward","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/prompted_output.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/prompted_output.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b55bd7c166344bfe"}},{"code_sha256_prefix":"7234e411db4436b8","entry":"ViltEmbeddings_prompted_forward","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/prompted_output.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/prompted_output.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7234e411db4436b8"}},{"code_sha256_prefix":"464c382329719d26","entry":"albef_prompted_forward","repo":"HaokunChen245/FedDAT","repo_kind":"official","path":"src/modeling/prompted_output.py","file_url":"https://github.com/HaokunChen245/FedDAT/blob/HEAD/src/modeling/prompted_output.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"464c382329719d26"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}