{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/critical-learning-periods-in-deep-neural","title":"Critical Learning Periods in Deep Neural Networks","arxiv_id":"1711.08856","date":"2017-11-24","proceeding":null,"authors":["Alessandro Achille","Matteo Rovere","Stefano Soatto"],"abstract":"Similar to humans and animals, deep artificial neural networks exhibit\ncritical periods during which a temporary stimulus deficit can impair the\ndevelopment of a skill. The extent of the impairment depends on the onset and\nlength of the deficit window, as in animal models, and on the size of the\nneural network. Deficits that do not affect low-level statistics, such as\nvertical flipping of the images, have no lasting effect on performance and can\nbe overcome with further training. To better understand this phenomenon, we use\nthe Fisher Information of the weights to measure the effective connectivity\nbetween layers of a network during training. Counterintuitively, information\nrises rapidly in the early phases of training, and then decreases, preventing\nredistribution of information resources in a phenomenon we refer to as a loss\nof \"Information Plasticity\". Our analysis suggests that the first few epochs\nare critical for the creation of strong connections that are optimal relative\nto the input data distribution. Once such strong connections are created, they\ndo not appear to change during additional training. These findings suggest that\nthe initial learning transient, under-scrutinized compared to asymptotic\nbehavior, plays a key role in determining the outcome of the training process.\nOur findings, combined with recent theoretical results in the literature, also\nsuggest that forgetting (decrease of information in the weights) is critical to\nachieving invariance and disentanglement in representation learning. Finally,\ncritical periods are not restricted to biological systems, but can emerge\nnaturally in learning systems, whether biological or artificial, due to\nfundamental constrains arising from learning dynamics and information\nprocessing.","url_abs":"http://arxiv.org/abs/1711.08856v3","url_pdf":"http://arxiv.org/pdf/1711.08856v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"critical-learning-periods-in-deep-neural","repo_url":"https://github.com/uw-mad-dash/Accordion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1711.08856","atlas_url":"https://app.syntology.ai/?focus=1711.08856","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}