{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parallelizing-legendre-memory-unit-training","title":"Parallelizing Legendre Memory Unit Training","arxiv_id":"2102.11417","date":"2021-02-22","proceeding":null,"authors":["Narsimha Chilkuri","Chris Eliasmith"],"abstract":"Recently, a new recurrent neural network (RNN) named the Legendre Memory Unit (LMU) was proposed and shown to achieve state-of-the-art performance on several benchmark datasets. Here we leverage the linear time-invariant (LTI) memory component of the LMU to construct a simplified variant that can be parallelized during training (and yet executed as an RNN during inference), thus overcoming a well known limitation of training RNNs on GPUs. We show that this reformulation that aids parallelizing, which can be applied generally to any deep network whose recurrent components are linear, makes training up to 200 times faster. Second, to validate its utility, we compare its performance against the original LMU and a variety of published LSTM and transformer networks on seven benchmarks, ranging from psMNIST to sentiment analysis to machine translation. We demonstrate that our models exhibit superior performance on all datasets, often using fewer parameters. For instance, our LMU sets a new state-of-the-art result on psMNIST, and uses half the parameters while outperforming DistilBERT and LSTM models on IMDB sentiment analysis.","url_abs":"https://arxiv.org/abs/2102.11417v2","url_pdf":"https://arxiv.org/pdf/2102.11417v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parallelizing-legendre-memory-unit-training","repo_url":"https://github.com/elisejiuqizhang/jax-lmu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"unanswered"}},{"paper_slug":"parallelizing-legendre-memory-unit-training","repo_url":"https://github.com/hrshtv/pytorch-lmu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"sequential-image-classification","task_name":"Sequential Image Classification"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"distilbert","method_name":"DistilBERT"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lmu","method_name":"LMU"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/sentiment-analysis-on-imdb","task":"Sentiment Analysis","dataset":"IMDb","model":"Modified LMU (34M)","rank_in_archive_order":27,"of":49,"metrics":{"Accuracy":"93.20"},"uses_additional_data":false},{"leaderboard":"/sota/sequential-image-classification-on-sequential","task":"Sequential Image Classification","dataset":"Sequential MNIST","model":"Modified LMU (165k)","rank_in_archive_order":6,"of":30,"metrics":{"Permuted Accuracy":"98.49%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.11417","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}