{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/disentangling-and-mitigating-the-impact-of","title":"Disentangling and Mitigating the Impact of Task Similarity for Continual Learning","arxiv_id":"2405.20236","date":"2024-05-30","proceeding":null,"authors":["Naoki Hiratani"],"abstract":"Continual learning of partially similar tasks poses a challenge for artificial neural networks, as task similarity presents both an opportunity for knowledge transfer and a risk of interference and catastrophic forgetting. However, it remains unclear how task similarity in input features and readout patterns influences knowledge transfer and forgetting, as well as how they interact with common algorithms for continual learning. Here, we develop a linear teacher-student model with latent structure and show analytically that high input feature similarity coupled with low readout similarity is catastrophic for both knowledge transfer and retention. Conversely, the opposite scenario is relatively benign. Our analysis further reveals that task-dependent activity gating improves knowledge retention at the expense of transfer, while task-dependent plasticity gating does not affect either retention or transfer performance at the over-parameterized limit. In contrast, weight regularization based on the Fisher information metric significantly improves retention, regardless of task similarity, without compromising transfer performance. Nevertheless, its diagonal approximation and regularization in the Euclidean space are much less robust against task similarity. We demonstrate consistent results in a permuted MNIST task with latent variables. Overall, this work provides insights into when continual learning is difficult and how to mitigate it.","url_abs":"https://arxiv.org/abs/2405.20236v1","url_pdf":"https://arxiv.org/pdf/2405.20236v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"disentangling-and-mitigating-the-impact-of","repo_url":"https://github.com/nhiratani/transfer_retention_model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":null,"task_name":"Permuted-MNIST"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nhiratani/transfer_retention_model","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"424259493a95cf3b","entry":"generate_Bs","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"pm/pm_data.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/pm/pm_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424259493a95cf3b"}},{"code_sha256_prefix":"d2dd90bcc1f2fb46","entry":"generate_g","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"teacher-student/lst_model.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/teacher-student/lst_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2dd90bcc1f2fb46"}},{"code_sha256_prefix":"c7a5e1caefa4e4ad","entry":"load_datafile","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"pm/pm_data.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/pm/pm_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7a5e1caefa4e4ad"}},{"code_sha256_prefix":"fb054362bc743632","entry":"y_latent","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"pm/pm_data.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/pm/pm_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb054362bc743632"}},{"code_sha256_prefix":"c1d73c67c7e036cb","entry":"generate_AB","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"teacher-student/lst_model.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/teacher-student/lst_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1d73c67c7e036cb"}},{"code_sha256_prefix":"97f98218a0447845","entry":"generate_AB_mix","repo":"nhiratani/transfer_retention_model","repo_kind":"official","path":"teacher-student/lst_model.py","file_url":"https://github.com/nhiratani/transfer_retention_model/blob/HEAD/teacher-student/lst_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97f98218a0447845"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}