{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-the-role-of-training-regimes-in","title":"Understanding the Role of Training Regimes in Continual Learning","arxiv_id":"2006.06958","date":"2020-06-12","proceeding":"NeurIPS 2020 12","authors":["Seyed Iman Mirzadeh","Mehrdad Farajtabar","Razvan Pascanu","Hassan Ghasemzadeh"],"abstract":"Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.","url_abs":"https://arxiv.org/abs/2006.06958v1","url_pdf":"https://arxiv.org/pdf/2006.06958v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-the-role-of-training-regimes-in","repo_url":"https://github.com/imirzadeh/stable-continual-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"understanding-the-role-of-training-regimes-in","repo_url":"https://github.com/MehdiAbbanaBennani/continual-learning-ogdplus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"understanding-the-role-of-training-regimes-in","repo_url":"https://github.com/chandar-lab/CMAL_Hanabi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-role-of-training-regimes-in","repo_url":"https://github.com/chandar-lab/Lifelong-Hanabi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.06958","atlas_url":"https://app.syntology.ai/?focus=2006.06958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.06958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chandar-lab/Lifelong-Hanabi","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MehdiAbbanaBennani/continual-learning-ogdplus","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/imirzadeh/stable-continual-learning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chandar-lab/CMAL_Hanabi","reach":{"status":"unanswered"}}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":17},"by_repo_kind":{"official":{"samples":20,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"583f9780bdd00a45","entry":"conv3x3","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/models.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"583f9780bdd00a45"}},{"code_sha256_prefix":"4531aa012b415174","entry":"create_conv_layer","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4531aa012b415174"}},{"code_sha256_prefix":"f5ffbe87b7ec7cb6","entry":"create_fc_layer","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5ffbe87b7ec7cb6"}},{"code_sha256_prefix":"50bc0f11bc526034","entry":"ResNet18","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/models.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50bc0f11bc526034"}},{"code_sha256_prefix":"a96da1b71ad1a9ef","entry":"bias_variable","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/model/model.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/model/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a96da1b71ad1a9ef"}},{"code_sha256_prefix":"687d5d0cc60d8e5a","entry":"clone_variable_list","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"687d5d0cc60d8e5a"}},{"code_sha256_prefix":"c9a434336aceb23b","entry":"er_mem_update_hindsight","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/er_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/er_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9a434336aceb23b"}},{"code_sha256_prefix":"dbc9c0571f3244fc","entry":"get_permuted_mnist","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbc9c0571f3244fc"}},{"code_sha256_prefix":"193bb84a5b458862","entry":"get_permuted_mnist_tasks","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"193bb84a5b458862"}},{"code_sha256_prefix":"f3f11a0226a1466d","entry":"get_rotated_mnist","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3f11a0226a1466d"}},{"code_sha256_prefix":"e75a0fb0c18a11fe","entry":"image_scaling","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e75a0fb0c18a11fe"}},{"code_sha256_prefix":"adff890682647c1e","entry":"init_experiment","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adff890682647c1e"}},{"code_sha256_prefix":"5d5a3e18f8eedade","entry":"log_metrics","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"stable_sgd/utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/stable_sgd/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d5a3e18f8eedade"}},{"code_sha256_prefix":"5b6dddd0f1e4b887","entry":"random_crop_and_pad_image","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b6dddd0f1e4b887"}},{"code_sha256_prefix":"876c106a35f12456","entry":"random_horizontal_flip","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/data_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"876c106a35f12456"}},{"code_sha256_prefix":"f6363cf3d882696c","entry":"test_task_sequence","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/fc_mnist.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/fc_mnist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6363cf3d882696c"}},{"code_sha256_prefix":"e401eb8ff332ab22","entry":"update_reservior","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/er_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/er_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e401eb8ff332ab22"}},{"code_sha256_prefix":"30e387d6f5d1ab03","entry":"vgg_conv_layer","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/vgg_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/vgg_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30e387d6f5d1ab03"}},{"code_sha256_prefix":"dceae2df5c5c47ec","entry":"vgg_fc_layer","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/utils/vgg_utils.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/utils/vgg_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dceae2df5c5c47ec"}},{"code_sha256_prefix":"70fae4aed55bdadc","entry":"weight_variable","repo":"imirzadeh/stable-continual-learning","repo_kind":"official","path":"external_libs/continual_learning_algorithms/model/model.py","file_url":"https://github.com/imirzadeh/stable-continual-learning/blob/HEAD/external_libs/continual_learning_algorithms/model/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70fae4aed55bdadc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}