Papers › LSTM: A Search Space Odyssey

LSTM: A Search Space Odyssey

13 Mar 2015arXiv:1503.04069archive 2025-07-28

Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber

Several variants of the Long Short-Term Memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework. In total, we summarize the results of 5400 experimental runs (≈15 years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

17 repositories listed; official and paper-mentioned ones first.

RohanAsnani/capstone_project_machine_learning-nanodegree mentioned on GitHubtfSyntology: not harvested report
a-martyn/unet mentioned on GitHubtfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to U-Net: Convolutional Networks for Biomedical Image Segmentation (arXiv:1505.04597; that paper's own run record: 530 ran (of which 247 constructed an object rather than computing a result; 378 with no instrument failure: 18 honoured, 10 violated, 350 with no contract checked; 152 where Syntology's instrument failed) · 227 unverified); nothing checks that they implement this paper's method. report
alexandrujeman/dqn mentioned on GitHubpytorchSyntology: not harvested report
alexjeman/dqn mentioned on GitHubpytorchSyntology: not harvested report
disooqi/DNN mentioned on GitHubNOASSERTIONSyntology: not harvested report
disooqi/Natasy mentioned on GitHubNOASSERTIONSyntology: not harvested report
flukeskywalker/highway-networks mentioned on GitHubNOASSERTIONSyntology: not harvested report
hadi-gharibi/pytorch-lstm mentioned on GitHubpytorchSyntology: not harvested report
kevin28520/RNN mentioned on GitHubSyntology: not harvested report
sw-scot/foco-mlmis mentioned on GitHubSyntology: not harvested report
tmatha/lstm mentioned on GitHubtfMITSyntology: not harvested report
widmi/widis-lstm-tools mentioned on GitHubpytorchGPL-3.0Syntology: not harvested report
2023-MindSpore-1/ms-code-14 mindsporeSyntology: not harvested report
2023-MindSpore-1/ms-code-219 mindsporeSyntology: not harvested report
cui0523/Code6 mindsporeSyntology: not harvested report
yangyucheng000/Paper-2 mindsporeSyntology: not harvested report
yangyucheng000/papercode-2 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition... (arXiv:2202.10108; that paper's own run record: 22 ran (of which 8 constructed an object rather than computing a result; 16 with no instrument failure: 0 honoured, 0 violated, 16 with no contract checked; 6 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Handwriting RecognitionMusic ModelingSpeech Recognition

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

LSTMSigmoid ActivationTanh Activation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections