{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/regularizing-rnns-by-stabilizing-activations","title":"Regularizing RNNs by Stabilizing Activations","arxiv_id":"1511.08400","date":"2015-11-26","proceeding":null,"authors":["David Krueger","Roland Memisevic"],"abstract":"We stabilize the activations of Recurrent Neural Networks (RNNs) by\npenalizing the squared distance between successive hidden states' norms.\n  This penalty term is an effective regularizer for RNNs including LSTMs and\nIRNNs, improving performance on character-level language modeling and phoneme\nrecognition, and outperforming weight noise and dropout.\n  We achieve competitive performance (18.6\\% PER) on the TIMIT phoneme\nrecognition task for RNNs evaluated without beam search or an RNN transducer.\n  With this penalty term, IRNN can achieve similar performance to LSTM on\nlanguage modeling, although adding the penalty term to the LSTM results in\nsuperior performance.\n  Our penalty term also prevents the exponential growth of IRNN's activations\noutside of their training horizon, allowing them to generalize to much longer\nsequences.","url_abs":"http://arxiv.org/abs/1511.08400v7","url_pdf":"http://arxiv.org/pdf/1511.08400v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"regularizing-rnns-by-stabilizing-activations","repo_url":"https://github.com/vimarshc/fastai_experiments","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"phoneme-recognition","task_name":"Phoneme Recognition"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.08400","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}