{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reversible-recurrent-neural-networks","title":"Reversible Recurrent Neural Networks","arxiv_id":"1810.10999","date":"2018-10-25","proceeding":"NeurIPS 2018 12","authors":["Matthew MacKay","Paul Vicol","Jimmy Ba","Roger Grosse"],"abstract":"Recurrent neural networks (RNNs) provide state-of-the-art performance in\nprocessing sequential data but are memory intensive to train, limiting the\nflexibility of RNN models which can be trained. Reversible RNNs---RNNs for\nwhich the hidden-to-hidden transition can be reversed---offer a path to reduce\nthe memory requirements of training, as hidden states need not be stored and\ninstead can be recomputed during backpropagation. We first show that perfectly\nreversible RNNs, which require no storage of the hidden activations, are\nfundamentally limited because they cannot forget information from their hidden\nstate. We then provide a scheme for storing a small number of bits in order to\nallow perfect reversal with forgetting. Our method achieves comparable\nperformance to traditional models while reducing the activation memory cost by\na factor of 10--15. We extend our technique to attention-based\nsequence-to-sequence models, where it maintains performance while reducing\nactivation memory cost by a factor of 5--10 in the encoder, and a factor of\n10--15 in the decoder.","url_abs":"http://arxiv.org/abs/1810.10999v1","url_pdf":"http://arxiv.org/pdf/1810.10999v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reversible-recurrent-neural-networks","repo_url":"https://github.com/matthewjmackay/reversible-rnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.10999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.10999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/matthewjmackay/reversible-rnn","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16739c192dffd2a8","entry":"convert_to_fixed","repo":"matthewjmackay/reversible-rnn","repo_kind":"official","path":"fixed_util.py","file_url":"https://github.com/matthewjmackay/reversible-rnn/blob/HEAD/fixed_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16739c192dffd2a8"}},{"code_sha256_prefix":"a85c5d616d8d068a","entry":"convert_to_float","repo":"matthewjmackay/reversible-rnn","repo_kind":"official","path":"fixed_util.py","file_url":"https://github.com/matthewjmackay/reversible-rnn/blob/HEAD/fixed_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a85c5d616d8d068a"}},{"code_sha256_prefix":"4ed5e00e94ee3740","entry":"embedded_dropout","repo":"matthewjmackay/reversible-rnn","repo_kind":"official","path":"language_modelling/embed_regularize.py","file_url":"https://github.com/matthewjmackay/reversible-rnn/blob/HEAD/language_modelling/embed_regularize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ed5e00e94ee3740"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}