{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rosa-accurate-parameter-efficient-fine-tuning","title":"RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation","arxiv_id":"2401.04679","date":"2024-01-09","proceeding":null,"authors":["Mahdi Nikdan","Soroush Tabesh","Elvir Crnčević","Dan Alistarh"],"abstract":"We investigate parameter-efficient fine-tuning (PEFT) methods that can provide good accuracy under limited computational and memory budgets in the context of large language models (LLMs). We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis that jointly trains $\\textit{low-rank}$ and $\\textit{highly-sparse}$ components on top of a set of fixed pretrained weights to efficiently approximate the performance of a full-fine-tuning (FFT) solution. Across a series of challenging generative tasks such as grade-school math and SQL query generation, which require fine-tuning for good performance, we show that RoSA outperforms LoRA, pure sparse fine-tuning, and alternative hybrid methods at the same parameter budget, and can even recover the performance of FFT on some tasks. We provide system support for RoSA to complement the training algorithm, specifically in the form of sparse GPU kernels which enable memory- and computationally-efficient training, and show that it is also compatible with low-precision base weights, resulting in the first joint representation combining quantization, low-rank and sparse approximations. Our code is available at https://github.com/IST-DASLab/RoSA.","url_abs":"https://arxiv.org/abs/2401.04679v7","url_pdf":"https://arxiv.org/pdf/2401.04679v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rosa-accurate-parameter-efficient-fine-tuning","repo_url":"https://github.com/ist-daslab/rosa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"rosa-accurate-parameter-efficient-fine-tuning","repo_url":"https://github.com/ist-daslab/panzamail","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"math","task_name":"Math"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.04679","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.04679"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ist-daslab/rosa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ist-daslab/panzamail","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"83e3dc312aa8844c","entry":"findattr","repo":"ist-daslab/rosa","repo_kind":"official","path":"llmfoundry/models/hf/hf_fsdp.py","file_url":"https://github.com/ist-daslab/rosa/blob/HEAD/llmfoundry/models/hf/hf_fsdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83e3dc312aa8844c"}},{"code_sha256_prefix":"7383beb2cec34bf1","entry":"rgetattr","repo":"ist-daslab/rosa","repo_kind":"official","path":"llmfoundry/models/hf/hf_fsdp.py","file_url":"https://github.com/ist-daslab/rosa/blob/HEAD/llmfoundry/models/hf/hf_fsdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7383beb2cec34bf1"}},{"code_sha256_prefix":"dd7e5308db9ee3c8","entry":"rhasattr","repo":"ist-daslab/rosa","repo_kind":"official","path":"llmfoundry/models/hf/hf_fsdp.py","file_url":"https://github.com/ist-daslab/rosa/blob/HEAD/llmfoundry/models/hf/hf_fsdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd7e5308db9ee3c8"}},{"code_sha256_prefix":"9b11c49c81759e69","entry":"compute_bleu_scores","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/interface/json.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/interface/json.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b11c49c81759e69"}},{"code_sha256_prefix":"c8e2cdc8524553df","entry":"compute_rouge_scores","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/interface/json.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/interface/json.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8e2cdc8524553df"}},{"code_sha256_prefix":"b73f5e72516aa64d","entry":"extract_only_plain_text","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/data_preparation/extract_emails.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/data_preparation/extract_emails.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b73f5e72516aa64d"}},{"code_sha256_prefix":"1ae0a808c04cb460","entry":"remove_date_time","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/data_preparation/extract_emails.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/data_preparation/extract_emails.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ae0a808c04cb460"}},{"code_sha256_prefix":"7b217d56ba2c03a1","entry":"retrieve_similar_emails","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/data_preparation/prepare_raft_emails.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/data_preparation/prepare_raft_emails.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b217d56ba2c03a1"}},{"code_sha256_prefix":"d290e0665d9f698d","entry":"skip_forwarded_messages","repo":"ist-daslab/panzamail","repo_kind":"listed","path":"src/panza/data_preparation/extract_emails.py","file_url":"https://github.com/ist-daslab/panzamail/blob/HEAD/src/panza/data_preparation/extract_emails.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d290e0665d9f698d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}