{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradient-projection-memory-for-continual-1","title":"Gradient Projection Memory for Continual Learning","arxiv_id":"2103.09762","date":"2021-03-17","proceeding":"ICLR 2021 1","authors":["Gobinda Saha","Isha Garg","Kaushik Roy"],"abstract":"The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.","url_abs":"https://arxiv.org/abs/2103.09762v1","url_pdf":"https://arxiv.org/pdf/2103.09762v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradient-projection-memory-for-continual-1","repo_url":"https://github.com/sahagobinda/GPM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.09762","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.09762"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sahagobinda/GPM","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e71c2a2bc2fc8909","entry":"compute_conv_output_size","repo":"sahagobinda/GPM","repo_kind":"official","path":"main_cifar100.py","file_url":"https://github.com/sahagobinda/GPM/blob/HEAD/main_cifar100.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e71c2a2bc2fc8909"}},{"code_sha256_prefix":"ebd151f2b9fc07af","entry":"get_model","repo":"sahagobinda/GPM","repo_kind":"official","path":"main_cifar100.py","file_url":"https://github.com/sahagobinda/GPM/blob/HEAD/main_cifar100.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ebd151f2b9fc07af"}},{"code_sha256_prefix":"52f190e75c2b2866","entry":"test","repo":"sahagobinda/GPM","repo_kind":"official","path":"main_cifar100.py","file_url":"https://github.com/sahagobinda/GPM/blob/HEAD/main_cifar100.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52f190e75c2b2866"}},{"code_sha256_prefix":"6e3c5f8afe521af7","entry":"update_GPM","repo":"sahagobinda/GPM","repo_kind":"official","path":"main_cifar100.py","file_url":"https://github.com/sahagobinda/GPM/blob/HEAD/main_cifar100.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e3c5f8afe521af7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}