{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multiplicative-lstm-for-sequence-modelling","title":"Multiplicative LSTM for sequence modelling","arxiv_id":"1609.07959","date":"2016-09-26","proceeding":null,"authors":["Ben Krause","Liang Lu","Iain Murray","Steve Renals"],"abstract":"We introduce multiplicative LSTM (mLSTM), a recurrent neural network\narchitecture for sequence modelling that combines the long short-term memory\n(LSTM) and multiplicative recurrent neural network architectures. mLSTM is\ncharacterised by its ability to have different recurrent transition functions\nfor each possible input, which we argue makes it more expressive for\nautoregressive density estimation. We demonstrate empirically that mLSTM\noutperforms standard LSTM and its deep variants for a range of character level\nlanguage modelling tasks. In this version of the paper, we regularise mLSTM to\nachieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also\napply a purely byte-level mLSTM on the WikiText-2 dataset to achieve a\ncharacter level entropy of 1.26 bits/char, corresponding to a word level\nperplexity of 88.8, which is comparable to word level LSTMs regularised in\nsimilar ways on the same task.","url_abs":"http://arxiv.org/abs/1609.07959v3","url_pdf":"http://arxiv.org/pdf/1609.07959v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multiplicative-lstm-for-sequence-modelling","repo_url":"https://github.com/astakara48/python_project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"density-estimation","task_name":"Density Estimation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"},{"method_slug":"weight-normalization","method_name":"Weight Normalization"},{"method_slug":"mlstm","method_name":"mLSTM"}],"datasets_introduced":[],"methods_introduced":[{"slug":"mlstm","name":"mLSTM","full_name":"Multiplicative LSTM"}],"results":[{"leaderboard":"/sota/language-modelling-on-hutter-prize","task":"Language Modelling","dataset":"Hutter Prize","model":"Large mLSTM +emb +WN +VD","rank_in_archive_order":14,"of":18,"metrics":{"Bit per Character (BPC)":"1.24","Number of params":"46M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"Large mLSTM +emb +WN +VD","rank_in_archive_order":18,"of":24,"metrics":{"Bit per Character (BPC)":"1.27","Number of params":"45M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"Unregularised mLSTM","rank_in_archive_order":21,"of":24,"metrics":{"Bit per Character (BPC)":"1.40","Number of params":"45M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"Large mLSTM","rank_in_archive_order":34,"of":42,"metrics":{"Bit per Character (BPC)":"1.24","Number of params":"46M"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1609.07959","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}