{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reviving-and-improving-recurrent-back","title":"Reviving and Improving Recurrent Back-Propagation","arxiv_id":"1803.06396","date":"2018-03-16","proceeding":"ICML 2018 7","authors":["Renjie Liao","Yuwen Xiong","Ethan Fetaya","Lisa Zhang","KiJung Yoon","Xaq Pitkow","Raquel Urtasun","Richard Zemel"],"abstract":"In this paper, we revisit the recurrent back-propagation (RBP) algorithm, discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neumann-RBP). We further investigate the relationship between Neumann-RBP and back propagation through time (BPTT) and its truncated version (TBPTT). Our Neumann-RBP has the same time complexity as TBPTT but only requires constant memory, whereas TBPTT's memory cost scales linearly with the number of truncation steps. We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks. All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are efficient and effective for optimizing convergent recurrent neural networks. Code is released at: \\url{https://github.com/lrjconan/RBP}.","url_abs":"https://arxiv.org/abs/1803.06396v4","url_pdf":"https://arxiv.org/pdf/1803.06396v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reviving-and-improving-recurrent-back","repo_url":"https://github.com/lrjconan/RBP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"document-classification","task_name":"Document Classification"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.06396","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.06396"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lrjconan/RBP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3ed1406329714c3f","entry":"cat_list_to_tensor","repo":"lrjconan/RBP","repo_kind":"official","path":"utils/model_helper.py","file_url":"https://github.com/lrjconan/RBP/blob/HEAD/utils/model_helper.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ed1406329714c3f"}},{"code_sha256_prefix":"7cfbfdb527b1aff5","entry":"conjugate_gradient","repo":"lrjconan/RBP","repo_kind":"official","path":"utils/model_helper.py","file_url":"https://github.com/lrjconan/RBP/blob/HEAD/utils/model_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cfbfdb527b1aff5"}},{"code_sha256_prefix":"68cc639edfbe703f","entry":"detach_param_with_grad","repo":"lrjconan/RBP","repo_kind":"official","path":"utils/model_helper.py","file_url":"https://github.com/lrjconan/RBP/blob/HEAD/utils/model_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68cc639edfbe703f"}},{"code_sha256_prefix":"d19a82fb4372e766","entry":"load_data","repo":"lrjconan/RBP","repo_kind":"official","path":"dataset/hypergrad_mnist.py","file_url":"https://github.com/lrjconan/RBP/blob/HEAD/dataset/hypergrad_mnist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d19a82fb4372e766"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}