{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neurogenesis-inspired-dictionary-learning","title":"Neurogenesis-Inspired Dictionary Learning: Online Model Adaption in a Changing World","arxiv_id":"1701.06106","date":"2017-01-22","proceeding":null,"authors":["Sahil Garg","Irina Rish","Guillermo Cecchi","Aurelie Lozano"],"abstract":"In this paper, we focus on online representation learning in non-stationary\nenvironments which may require continuous adaptation of model architecture. We\npropose a novel online dictionary-learning (sparse-coding) framework which\nincorporates the addition and deletion of hidden units (dictionary elements),\nand is inspired by the adult neurogenesis phenomenon in the dentate gyrus of\nthe hippocampus, known to be associated with improved cognitive function and\nadaptation to new environments. In the online learning setting, where new input\ninstances arrive sequentially in batches, the neuronal-birth is implemented by\nadding new units with random initial weights (random dictionary elements); the\nnumber of new units is determined by the current performance (representation\nerror) of the dictionary, higher error causing an increase in the birth rate.\nNeuronal-death is implemented by imposing l1/l2-regularization (group sparsity)\non the dictionary within the block-coordinate descent optimization at each\niteration of our online alternating minimization scheme, which iterates between\nthe code and dictionary updates. Finally, hidden unit connectivity adaptation\nis facilitated by introducing sparsity in dictionary elements. Our empirical\nevaluation on several real-life datasets (images and language) as well as on\nsynthetic data demonstrates that the proposed approach can considerably\noutperform the state-of-art fixed-size (nonadaptive) online sparse coding of\nMairal et al. (2009) in the presence of nonstationary data. Moreover, we\nidentify certain properties of the data (e.g., sparse inputs with nearly\nnon-overlapping supports) and of the model (e.g., dictionary sparsity)\nassociated with such improvements.","url_abs":"http://arxiv.org/abs/1701.06106v2","url_pdf":"http://arxiv.org/pdf/1701.06106v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neurogenesis-inspired-dictionary-learning","repo_url":"https://github.com/sgarg87/neurogenesis_inspired_dictionary_learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"dictionary-learning","task_name":"Dictionary Learning"},{"task_slug":null,"task_name":"Hippocampus"},{"task_slug":"l2-regularization","task_name":"L2 Regularization"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}