{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-quadratic-regularizers-explicit","title":"How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation","arxiv_id":"2102.02805","date":"2021-02-04","proceeding":null,"authors":["Ekdeep Singh Lubana","Puja Trivedi","Danai Koutra","Robert P. Dick"],"abstract":"Catastrophic forgetting undermines the effectiveness of deep neural networks (DNNs) in scenarios such as continual learning and lifelong learning. While several methods have been proposed to tackle this problem, there is limited work explaining why these methods work well. This paper has the goal of better explaining a popularly used technique for avoiding catastrophic forgetting: quadratic regularization. We show that quadratic regularizers prevent forgetting of past tasks by interpolating current and previous values of model parameters at every training iteration. Over multiple training iterations, this interpolation operation reduces the learning rates of more important model parameters, thereby minimizing their movement. Our analysis also reveals two drawbacks of quadratic regularization: (a) dependence of parameter interpolation on training hyperparameters, which often leads to training instability and (b) assignment of lower importance to deeper layers, which are generally the place forgetting occurs in DNNs. Via a simple modification to the order of operations, we show these drawbacks can be easily avoided, resulting in 6.2\\% higher average accuracy at 4.5\\% lower average forgetting. We confirm the robustness of our results by training over 2000 models in different settings. Code available at \\url{https://github.com/EkdeepSLubana/QRforgetting}","url_abs":"https://arxiv.org/abs/2102.02805v5","url_pdf":"https://arxiv.org/pdf/2102.02805v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-quadratic-regularizers-explicit","repo_url":"https://github.com/EkdeepSLubana/EMR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"rethinking-quadratic-regularizers-explicit","repo_url":"https://github.com/EkdeepSLubana/QRforgetting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"lifelong-learning","task_name":"Lifelong learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.02805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.02805"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EkdeepSLubana/EMR","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EkdeepSLubana/QRforgetting","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"70677cb9f11d213a","entry":"clone_model_grads","repo":"EkdeepSLubana/QRforgetting","repo_kind":"official","path":"opt_based.py","file_url":"https://github.com/EkdeepSLubana/QRforgetting/blob/HEAD/opt_based.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70677cb9f11d213a"}},{"code_sha256_prefix":"d41112046e6c9be7","entry":"clone_model_params","repo":"EkdeepSLubana/QRforgetting","repo_kind":"official","path":"opt_based.py","file_url":"https://github.com/EkdeepSLubana/QRforgetting/blob/HEAD/opt_based.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d41112046e6c9be7"}},{"code_sha256_prefix":"4755d06884e9cce9","entry":"normalize","repo":"EkdeepSLubana/QRforgetting","repo_kind":"official","path":"reg_based.py","file_url":"https://github.com/EkdeepSLubana/QRforgetting/blob/HEAD/reg_based.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4755d06884e9cce9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}