{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parameter-efficient-quasi-orthogonal-fine","title":"Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation","arxiv_id":"2404.04316","date":"2024-04-05","proceeding":null,"authors":["Xinyu Ma","Xu Chu","Zhibang Yang","Yang Lin","Xin Gao","Junfeng Zhao"],"abstract":"With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\\mathcal{O}(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $\\mathcal{O}(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $\\mathcal{O}(d^2)$ to $\\mathcal{O}(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.","url_abs":"https://arxiv.org/abs/2404.04316v2","url_pdf":"https://arxiv.org/pdf/2404.04316v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parameter-efficient-quasi-orthogonal-fine","repo_url":"https://github.com/arthurleom/peft-givens","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"parameter-efficient-quasi-orthogonal-fine","repo_url":"https://github.com/hiyouga/llama-factory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.04316","atlas_url":"https://app.syntology.ai/?focus=2404.04316","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.04316"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ArthurLeoM/peft-givens","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arthurleom/peft-givens","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hiyouga/llama-factory","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"93f1b05cd20c2ff6","entry":"GivensLayer","repo":"arthurleom/peft-givens","repo_kind":"official","path":"tuners/givens/layer.py","file_url":"https://github.com/arthurleom/peft-givens/blob/HEAD/tuners/givens/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93f1b05cd20c2ff6"}},{"code_sha256_prefix":"6a1220317ec3675f","entry":"check_if_peft_model","repo":"ArthurLeoM/peft-givens","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/ArthurLeoM/peft-givens/blob/HEAD/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a1220317ec3675f"}},{"code_sha256_prefix":"a3daaf3e5bdfc709","entry":"is_torch_tpu_available","repo":"ArthurLeoM/peft-givens","repo_kind":"official","path":"import_utils.py","file_url":"https://github.com/ArthurLeoM/peft-givens/blob/HEAD/import_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3daaf3e5bdfc709"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}