{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrent-neural-networks-vanishing-and","title":"Recurrent neural networks: vanishing and exploding gradients are not the end of the story","arxiv_id":"2405.21064","date":"2024-05-31","proceeding":null,"authors":["Nicolas Zucchet","Antonio Orvieto"],"abstract":"Recurrent neural networks (RNNs) notoriously struggle to learn long-term memories, primarily due to vanishing and exploding gradients. The recent success of state-space models (SSMs), a subclass of RNNs, to overcome such difficulties challenges our theoretical understanding. In this paper, we delve into the optimization challenges of RNNs and discover that, as the memory of a network increases, changes in its parameters result in increasingly large output variations, making gradient-based learning highly sensitive, even without exploding gradients. Our analysis further reveals the importance of the element-wise recurrence design pattern combined with careful parametrizations in mitigating this effect. This feature is present in SSMs, as well as in other architectures, such as LSTMs. Overall, our insights provide a new explanation for some of the difficulties in gradient-based learning of RNNs and why some architectures perform better than others.","url_abs":"https://arxiv.org/abs/2405.21064v2","url_pdf":"https://arxiv.org/pdf/2405.21064v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrent-neural-networks-vanishing-and","repo_url":"https://github.com/NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.21064","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.21064"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/NicolasZucchet/Vanishing-andexploding-gradients-are-not-the-end-of-the-story","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"0212f0ba3cfe6d69","entry":"binary_operator_diag","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0212f0ba3cfe6d69"}},{"code_sha256_prefix":"76a18d8aa82c03b6","entry":"matrix_init","repo":"NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story","repo_kind":"official","path":"source/model.py","file_url":"https://github.com/NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story/blob/HEAD/source/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76a18d8aa82c03b6"}},{"code_sha256_prefix":"4abcfdeb27acbd8a","entry":"nu_init","repo":"NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story","repo_kind":"official","path":"source/model.py","file_url":"https://github.com/NicolasZucchet/Vanishing-and-exploding-gradients-are-not-the-end-of-the-story/blob/HEAD/source/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4abcfdeb27acbd8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}