{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/one-initialization-to-rule-them-all-fine","title":"Parameter Efficient Fine-tuning via Explained Variance Adaptation","arxiv_id":"2410.07170","date":"2024-10-09","proceeding":null,"authors":["Fabian Paischer","Lukas Hauzenberger","Thomas Schmied","Benedikt Alkin","Marc Peter Deisenroth","Sepp Hochreiter"],"abstract":"Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weights via low-rank adaptation (LoRA). Existing initialization strategies for LoRA often rely on singular value decompositions (SVD) of gradients or weight matrices. However, they do not provably maximize the expected gradient signal, which is critical for fast adaptation. To this end, we introduce Explained Variance Adaptation (EVA), an initialization scheme that uses the directions capturing the most activation variance, provably maximizing the expected gradient signal and accelerating fine-tuning. EVA performs incremental SVD on minibatches of activation vectors and selects the right-singular vectors for initialization once they converged. Further, by selecting the directions that capture the most activation-variance for a given rank budget, EVA accommodates adaptive ranks that reduce the number of trainable parameters, while maintaining or improving downstream performance. We apply EVA to a variety of fine-tuning tasks as language generation and understanding, image classification, and reinforcement learning. EVA exhibits faster convergence than competitors and achieves the highest average score across a multitude of tasks per domain while reducing the number of trainable parameters through rank redistribution.","url_abs":"https://arxiv.org/abs/2410.07170v4","url_pdf":"https://arxiv.org/pdf/2410.07170v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"one-initialization-to-rule-them-all-fine","repo_url":"https://github.com/BenediktAlkin/vtab1k-pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"one-initialization-to-rule-them-all-fine","repo_url":"https://github.com/ml-jku/EVA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.07170","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07170"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/EVA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BenediktAlkin/vtab1k-pytorch","reach":null}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1da02603acd65cf8","entry":"_BufferedIncrementalPCA","repo":"BenediktAlkin/vtab1k-pytorch","repo_kind":"official","path":"src/vtab/adapters/eva_linear.py","file_url":"https://github.com/BenediktAlkin/vtab1k-pytorch/blob/HEAD/src/vtab/adapters/eva_linear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1da02603acd65cf8"}},{"code_sha256_prefix":"66d79fedd060dc95","entry":"get_eva_state_dict","repo":"ml-jku/EVA","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ml-jku/EVA/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66d79fedd060dc95"}},{"code_sha256_prefix":"d82feb60c9bc01b7","entry":"match_module_name","repo":"ml-jku/EVA","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ml-jku/EVA/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d82feb60c9bc01b7"}},{"code_sha256_prefix":"9ca1d5db1b9dc592","entry":"EvaLinear","repo":"BenediktAlkin/vtab1k-pytorch","repo_kind":"official","path":"src/vtab/adapters/eva_linear.py","file_url":"https://github.com/BenediktAlkin/vtab1k-pytorch/blob/HEAD/src/vtab/adapters/eva_linear.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ca1d5db1b9dc592"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}