{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/federated-learning-from-vision-language","title":"Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method","arxiv_id":"2409.19610","date":"2024-09-29","proceeding":null,"authors":["Bikang Pan","Wei Huang","Ye Shi"],"abstract":"Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.","url_abs":"https://arxiv.org/abs/2409.19610v1","url_pdf":"https://arxiv.org/pdf/2409.19610v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"federated-learning-from-vision-language","repo_url":"https://github.com/PanBikang/PromptFolio","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"federated-learning","task_name":"Federated Learning"},{"task_slug":"learning-theory","task_name":"Learning Theory"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"portfolio-optimization","task_name":"Portfolio Optimization"},{"task_slug":"prompt-learning","task_name":"Prompt Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.19610","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.19610"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PanBikang/PromptFolio","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":3,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ab2485d028020739","entry":"average_weights","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"utils.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab2485d028020739"}},{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"697e29e1bbc08d3d","entry":"average_weights","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"fed_utils.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/fed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"697e29e1bbc08d3d"}},{"code_sha256_prefix":"e2f5c1cc82e9df78","entry":"build_model","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2f5c1cc82e9df78"}},{"code_sha256_prefix":"a064a0985b120c40","entry":"cluster_weights","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"fed_utils.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/fed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a064a0985b120c40"}},{"code_sha256_prefix":"fbf8c0143d9c48e3","entry":"load","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbf8c0143d9c48e3"}},{"code_sha256_prefix":"49d5e536fa27cddc","entry":"record_net_data_stats","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"datasplit.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/datasplit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49d5e536fa27cddc"}},{"code_sha256_prefix":"03564ffca35b29ce","entry":"renormalize","repo":"PanBikang/PromptFolio","repo_kind":"official","path":"datasplit.py","file_url":"https://github.com/PanBikang/PromptFolio/blob/HEAD/datasplit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03564ffca35b29ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}