{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-theoretically-grounded-application-of","title":"A Theoretically Grounded Application of Dropout in Recurrent Neural Networks","arxiv_id":"1512.05287","date":"2015-12-16","proceeding":"NeurIPS 2016 12","authors":["Yarin Gal","Zoubin Ghahramani"],"abstract":"Recurrent neural networks (RNNs) stand at the forefront of many recent\ndevelopments in deep learning. Yet a major difficulty with these models is\ntheir tendency to overfit, with dropout shown to fail when applied to recurrent\nlayers. Recent results at the intersection of Bayesian modelling and deep\nlearning offer a Bayesian interpretation of common deep learning techniques\nsuch as dropout. This grounding of dropout in approximate Bayesian inference\nsuggests an extension of the theoretical results, offering insights into the\nuse of dropout with RNN models. We apply this new variational inference based\ndropout technique in LSTM and GRU models, assessing it on language modelling\nand sentiment analysis tasks. The new approach outperforms existing techniques,\nand to the best of our knowledge improves on the single model state-of-the-art\nin language modelling with the Penn Treebank (73.4 test perplexity). This\nextends our arsenal of variational tools in deep learning.","url_abs":"http://arxiv.org/abs/1512.05287v5","url_pdf":"http://arxiv.org/pdf/1512.05287v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/HKUST-KnowComp/R-Net","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/SuperKam91/bnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/Waino/hnmt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/abdelrahmansaud/vLSTM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/bhaddow/dev-nematus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/jiahuei/COMIC-Compact-Image-Captioning-with-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/jiahuei/COMIC-Towards-A-Compact-Image-Captioning-Model-with-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/josephdviviano/lstm-variational-dropout","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/magahub/songrnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/martin-gorner/tensorflow-rnn-shakespeare","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/samihadouaj/siyanWork","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/shivam13juna/Sequence_Prediction_LSTM_CHAR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/yaringal/BayesianRNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-theoretically-grounded-application-of","repo_url":"https://github.com/zhangyaoyuan/GAN-Simplification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"embedding-dropout","method_name":"Embedding Dropout"},{"method_slug":"gru","method_name":"GRU"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"}],"datasets_introduced":[],"methods_introduced":[{"slug":"embedding-dropout","name":"Embedding Dropout","full_name":"Embedding Dropout"},{"slug":"variational-dropout","name":"Variational Dropout","full_name":"Variational Dropout"}],"results":[{"leaderboard":"/sota/language-modelling-on-penn-treebank-word","task":"Language Modelling","dataset":"Penn Treebank (Word Level)","model":"Gal & Ghahramani (2016) - Variational LSTM (large)","rank_in_archive_order":35,"of":43,"metrics":{"Test perplexity":"75.2","Validation perplexity":"77.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-penn-treebank-word","task":"Language Modelling","dataset":"Penn Treebank (Word Level)","model":"Gal & Ghahramani (2016) - Variational LSTM (medium)","rank_in_archive_order":38,"of":43,"metrics":{"Test perplexity":"79.7","Validation perplexity":"81.9"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1512.05287","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1512.05287"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SuperKam91/bnn","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/samihadouaj/siyanWork","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Waino/hnmt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HKUST-KnowComp/R-Net","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhangyaoyuan/GAN-Simplification","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/martin-gorner/tensorflow-rnn-shakespeare","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/josephdviviano/lstm-variational-dropout","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shivam13juna/Sequence_Prediction_LSTM_CHAR","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bhaddow/dev-nematus","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/magahub/songrnn","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaringal/BayesianRNN","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiahuei/COMIC-Compact-Image-Captioning-with-Attention","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiahuei/COMIC-Towards-A-Compact-Image-Captioning-Model-with-Attention","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abdelrahmansaud/vLSTM","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0c44c51d06ba92e6","entry":"convert_from_alphabet","repo":"martin-gorner/tensorflow-rnn-shakespeare","repo_kind":"listed","path":"my_txtutils.py","file_url":"https://github.com/martin-gorner/tensorflow-rnn-shakespeare/blob/HEAD/my_txtutils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c44c51d06ba92e6"}},{"code_sha256_prefix":"85e5d2ed0f8daaa1","entry":"convert_to_alphabet","repo":"martin-gorner/tensorflow-rnn-shakespeare","repo_kind":"listed","path":"my_txtutils.py","file_url":"https://github.com/martin-gorner/tensorflow-rnn-shakespeare/blob/HEAD/my_txtutils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85e5d2ed0f8daaa1"}},{"code_sha256_prefix":"89e3c1684fe53b8b","entry":"encode_text","repo":"martin-gorner/tensorflow-rnn-shakespeare","repo_kind":"listed","path":"my_txtutils.py","file_url":"https://github.com/martin-gorner/tensorflow-rnn-shakespeare/blob/HEAD/my_txtutils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89e3c1684fe53b8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}