{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/factorization-tricks-for-lstm-networks","title":"Factorization tricks for LSTM networks","arxiv_id":"1703.10722","date":"2017-03-31","proceeding":null,"authors":["Oleksii Kuchaiev","Boris Ginsburg"],"abstract":"We present two simple ways of reducing the number of parameters and\naccelerating the training of large Long Short-Term Memory (LSTM) networks: the\nfirst one is \"matrix factorization by design\" of LSTM matrix into the product\nof two smaller matrices, and the second one is partitioning of LSTM matrix, its\ninputs and states into the independent groups. Both approaches allow us to\ntrain large LSTM networks significantly faster to the near state-of the art\nperplexity while using significantly less RNN parameters.","url_abs":"http://arxiv.org/abs/1703.10722v3","url_pdf":"http://arxiv.org/pdf/1703.10722v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"factorization-tricks-for-lstm-networks","repo_url":"https://github.com/okuchaiev/f-lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"factorization-tricks-for-lstm-networks","repo_url":"https://github.com/rdspring1/PyTorch_GBW_LM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"BIG G-LSTM-2","rank_in_archive_order":22,"of":27,"metrics":{"PPL":"36.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.10722","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}