{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/orthogonal-subspace-learning-for-language","title":"Orthogonal Subspace Learning for Language Model Continual Learning","arxiv_id":"2310.14152","date":"2023-10-22","proceeding":null,"authors":["Xiao Wang","Tianze Chen","Qiming Ge","Han Xia","Rong Bao","Rui Zheng","Qi Zhang","Tao Gui","Xuanjing Huang"],"abstract":"Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.","url_abs":"https://arxiv.org/abs/2310.14152v1","url_pdf":"https://arxiv.org/pdf/2310.14152v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"orthogonal-subspace-learning-for-language","repo_url":"https://github.com/cmnfriend/o-lora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.14152","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.14152"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmnfriend/o-lora","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f82932ddd399c698","entry":"check_model","repo":"cmnfriend/o-lora","repo_kind":"official","path":"src/uie_collator.py","file_url":"https://github.com/cmnfriend/o-lora/blob/HEAD/src/uie_collator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f82932ddd399c698"}},{"code_sha256_prefix":"e168ce39d04b76f1","entry":"exact_match_score","repo":"cmnfriend/o-lora","repo_kind":"official","path":"src/compute_metrics.py","file_url":"https://github.com/cmnfriend/o-lora/blob/HEAD/src/compute_metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e168ce39d04b76f1"}},{"code_sha256_prefix":"9fd7486ecca551f7","entry":"gen_cache_path","repo":"cmnfriend/o-lora","repo_kind":"official","path":"src/uie_dataset_lora.py","file_url":"https://github.com/cmnfriend/o-lora/blob/HEAD/src/uie_dataset_lora.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fd7486ecca551f7"}},{"code_sha256_prefix":"52a9f448d8cbd82a","entry":"normalize_answer","repo":"cmnfriend/o-lora","repo_kind":"official","path":"src/compute_metrics.py","file_url":"https://github.com/cmnfriend/o-lora/blob/HEAD/src/compute_metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52a9f448d8cbd82a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}