{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/overcoming-catastrophic-forgetting-by-soft","title":"Overcoming Catastrophic Forgetting by Soft Parameter Pruning","arxiv_id":"1812.01640","date":"2018-12-04","proceeding":null,"authors":["Jian Peng","Jiang Hao","Zhuo Li","Enqiang Guo","Xiaohong Wan","Deng Min","Qing Zhu","Haifeng Li"],"abstract":"Catastrophic forgetting is a challenge issue in continual learning when a\ndeep neural network forgets the knowledge acquired from the former task after\nlearning on subsequent tasks. However, existing methods try to find the joint\ndistribution of parameters shared with all tasks. This idea can be questionable\nbecause this joint distribution may not present when the number of tasks\nincrease. On the other hand, It also leads to \"long-term\" memory issue when the\nnetwork capacity is limited since adding tasks will \"eat\" the network capacity.\nIn this paper, we proposed a Soft Parameters Pruning (SPP) strategy to reach\nthe trade-off between short-term and long-term profit of a learning model by\nfreeing those parameters less contributing to remember former task domain\nknowledge to learn future tasks, and preserving memories about previous tasks\nvia those parameters effectively encoding knowledge about tasks at the same\ntime. The SPP also measures the importance of parameters by information entropy\nin a label free manner. The experiments on several tasks shows SPP model\nachieved the best performance compared with others state-of-the-art methods.\nExperiment results also indicate that our method is less sensitive to\nhyper-parameter and better generalization. Our research suggests that a softer\nstrategy, i.e. approximate optimize or sub-optimal solution, will benefit\nalleviating the dilemma of memory. The source codes are available at\nhttps://github.com/lehaifeng/Learning_by_memory.","url_abs":"http://arxiv.org/abs/1812.01640v1","url_pdf":"http://arxiv.org/pdf/1812.01640v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"overcoming-catastrophic-forgetting-by-soft","repo_url":"https://github.com/lehaifeng/Learning_by_memory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}