{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimizing-reusable-knowledge-for-continual","title":"Optimizing Reusable Knowledge for Continual Learning via Metalearning","arxiv_id":"2106.05390","date":"2021-06-09","proceeding":"NeurIPS 2021 12","authors":["Julio Hurtado","Alain Raymond-Saez","Alvaro Soto"],"abstract":"When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting of old information. To address this issue, we propose MetA Reusable Knowledge or MARK, a new method that fosters weight reusability instead of overwriting when learning a new task. Specifically, MARK keeps a set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the 20-Split-MiniImageNet dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that is selectively used by each task.","url_abs":"https://arxiv.org/abs/2106.05390v3","url_pdf":"https://arxiv.org/pdf/2106.05390v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-reusable-knowledge-for-continual","repo_url":"https://github.com/JuliousHurtado/meta-training-setup","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.05390","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.05390"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JuliousHurtado/meta-training-setup","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5c3574e986df9984","entry":"calculate_md5","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"dataloaders/utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/dataloaders/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c3574e986df9984"}},{"code_sha256_prefix":"46b62ccc9e5b9878","entry":"check_integrity","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"dataloaders/utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/dataloaders/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46b62ccc9e5b9878"}},{"code_sha256_prefix":"b3cb500b2f04323d","entry":"check_md5","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"dataloaders/utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/dataloaders/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3cb500b2f04323d"}},{"code_sha256_prefix":"e71c2a2bc2fc8909","entry":"compute_conv_output_size","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"models/conv.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/models/conv.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e71c2a2bc2fc8909"}},{"code_sha256_prefix":"ed93b06c8ef9580a","entry":"get_diff_weights","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed93b06c8ef9580a"}},{"code_sha256_prefix":"6c81a9cc9e6a6dd2","entry":"init_grads_out","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c81a9cc9e6a6dd2"}},{"code_sha256_prefix":"a6d95a8ba8160eee","entry":"print_log_acc_bwt","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"utils.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6d95a8ba8160eee"}},{"code_sha256_prefix":"4e987939c21e03a4","entry":"test","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"approach.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/approach.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e987939c21e03a4"}},{"code_sha256_prefix":"6beaaddedad65e73","entry":"train_meta_batch","repo":"JuliousHurtado/meta-training-setup","repo_kind":"official","path":"approach.py","file_url":"https://github.com/JuliousHurtado/meta-training-setup/blob/HEAD/approach.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6beaaddedad65e73"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}