{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fedmllm-federated-fine-tuning-mllm-on","title":"FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data","arxiv_id":"2411.14717","date":"2024-11-22","proceeding":null,"authors":["Binqian Xu","Xiangbo Shu","Haiyang Mei","GuoSen Xie","Basura Fernando","Mike Zheng Shou","Jinhui Tang"],"abstract":"Multimodal Large Language Models (MLLMs) have made significant advancements, demonstrating powerful capabilities in processing and understanding multimodal data. Fine-tuning MLLMs with Federated Learning (FL) allows for expanding the training data scope by including private data sources, thereby enhancing their practical applicability in privacy-sensitive domains. However, current research remains in the early stage, particularly in addressing the \\textbf{multimodal heterogeneities} in real-world applications. In this paper, we introduce a benchmark for evaluating various downstream tasks in the federated fine-tuning of MLLMs within multimodal heterogeneous scenarios, laying the groundwork for the research in the field. Our benchmark encompasses two datasets, five comparison baselines, and four multimodal scenarios, incorporating over ten types of modal heterogeneities. To address the challenges posed by modal heterogeneity, we develop a general FedMLLM framework that integrates four representative FL methods alongside two modality-agnostic strategies. Extensive experimental results show that our proposed FL paradigm improves the performance of MLLMs by broadening the range of training data and mitigating multimodal heterogeneity. Code is available at https://github.com/1xbq1/FedMLLM","url_abs":"https://arxiv.org/abs/2411.14717v1","url_pdf":"https://arxiv.org/pdf/2411.14717v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fedmllm-federated-fine-tuning-mllm-on","repo_url":"https://github.com/1xbq1/fedmllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.14717","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.14717"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/1xbq1/fedmllm","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"1a3baf406bc7de64","entry":"get_parameter_number","repo":"1xbq1/fedmllm","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/finetune/finetune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a3baf406bc7de64"}},{"code_sha256_prefix":"4abcd67770ee2e3f","entry":"conversation_to_ids","repo":"1xbq1/fedmllm","repo_kind":"official","path":"finetune/dataset.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/finetune/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4abcd67770ee2e3f"}},{"code_sha256_prefix":"d6c1484ab50b4e26","entry":"conversation_to_ids_minicpm","repo":"1xbq1/fedmllm","repo_kind":"official","path":"finetune/dataset.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/finetune/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6c1484ab50b4e26"}},{"code_sha256_prefix":"f9c00b006d3afba3","entry":"cosine_warm_learning_rate","repo":"1xbq1/fedmllm","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9c00b006d3afba3"}},{"code_sha256_prefix":"4212780bf7da9ddd","entry":"data_collator","repo":"1xbq1/fedmllm","repo_kind":"official","path":"finetune/dataset.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/finetune/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4212780bf7da9ddd"}},{"code_sha256_prefix":"2478f896f262d2a9","entry":"meld_dump","repo":"1xbq1/fedmllm","repo_kind":"official","path":"eval_crisismmd.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/eval_crisismmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2478f896f262d2a9"}},{"code_sha256_prefix":"7008810ccfff7ac2","entry":"meld_dump","repo":"1xbq1/fedmllm","repo_kind":"official","path":"eval_hateful.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/eval_hateful.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7008810ccfff7ac2"}},{"code_sha256_prefix":"33f61f6a51218672","entry":"meld_dump","repo":"1xbq1/fedmllm","repo_kind":"official","path":"eval_hateful_prompt.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/eval_hateful_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33f61f6a51218672"}},{"code_sha256_prefix":"e01df950d91ba4b6","entry":"remove_url","repo":"1xbq1/fedmllm","repo_kind":"official","path":"eval_crisismmd.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/eval_crisismmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e01df950d91ba4b6"}},{"code_sha256_prefix":"32f8f1bcb6c46c13","entry":"save_result","repo":"1xbq1/fedmllm","repo_kind":"official","path":"eval_medical.py","file_url":"https://github.com/1xbq1/fedmllm/blob/HEAD/eval_medical.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32f8f1bcb6c46c13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}