{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-fast-dropout-and-its-applicability-to","title":"On Fast Dropout and its Applicability to Recurrent Networks","arxiv_id":"1311.0701","date":"2013-11-04","proceeding":null,"authors":["Justin Bayer","Christian Osendorfer","Daniela Korhammer","Nutan Chen","Sebastian Urban","Patrick van der Smagt"],"abstract":"Recurrent Neural Networks (RNNs) are rich models for the processing of\nsequential data. Recent work on advancing the state of the art has been focused\non the optimization or modelling of RNNs, mostly motivated by adressing the\nproblems of the vanishing and exploding gradients. The control of overfitting\nhas seen considerably less attention. This paper contributes to that by\nanalyzing fast dropout, a recent regularization method for generalized linear\nmodels and neural networks from a back-propagation inspired perspective. We\nshow that fast dropout implements a quadratic form of an adaptive,\nper-parameter regularizer, which rewards large weights in the light of\nunderfitting, penalizes them for overconfident predictions and vanishes at\nminima of an unregularized training loss. The derivatives of that regularizer\nare exclusively based on the training error signal. One consequence of this is\nthe absense of a global weight attractor, which is particularly appealing for\nRNNs, since the dynamics are not biased towards a certain regime. We positively\ntest the hypothesis that this improves the performance of RNNs on four musical\ndata sets.","url_abs":"http://arxiv.org/abs/1311.0701v7","url_pdf":"http://arxiv.org/pdf/1311.0701v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-fast-dropout-and-its-applicability-to","repo_url":"https://github.com/George091/RNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}