{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-activation-regularization-for","title":"Revisiting Activation Regularization for Language RNNs","arxiv_id":"1708.01009","date":"2017-08-03","proceeding":null,"authors":["Stephen Merity","Bryan McCann","Richard Socher"],"abstract":"Recurrent neural networks (RNNs) serve as a fundamental building block for\nmany sequence tasks across natural language processing. Recent research has\nfocused on recurrent dropout techniques or custom RNN cells in order to improve\nperformance. Both of these can require substantial modifications to the machine\nlearning model or to the underlying RNN configurations. We revisit traditional\nregularization techniques, specifically L2 regularization on RNN activations\nand slowness regularization over successive hidden states, to improve the\nperformance of RNNs on the task of language modeling. Both of these techniques\nrequire minimal modification to existing RNN architectures and result in\nperformance improvements comparable or superior to more complicated\nregularization techniques or custom cell architectures. These regularization\ntechniques can be used without any modification on optimized LSTM\nimplementations such as the NVIDIA cuDNN LSTM.","url_abs":"http://arxiv.org/abs/1708.01009v1","url_pdf":"http://arxiv.org/pdf/1708.01009v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"l2-regularization","task_name":"L2 Regularization"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"activation-regularization","method_name":"Activation Regularization"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"recurrent-dropout","method_name":"Recurrent Dropout"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"temporal-activation-regularization","method_name":"Temporal Activation Regularization"}],"datasets_introduced":[],"methods_introduced":[{"slug":"activation-regularization","name":"Activation Regularization","full_name":"Activation Regularization"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1708.01009","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}