{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptively-truncating-backpropagation-through","title":"Adaptively Truncating Backpropagation Through Time to Control Gradient Bias","arxiv_id":"1905.07473","date":"2019-05-17","proceeding":null,"authors":["Christopher Aicher","Nicholas J. Foti","Emily B. Fox"],"abstract":"Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge if the truncation length is too small, or will converge slowly if it is too large. We propose an adaptive TBPTT scheme that converts the problem from choosing a temporal lag to one of choosing a tolerable amount of gradient bias. For many realistic RNNs, the TBPTT gradients decay geometrically in expectation for large lags; under this condition, we can control the bias by varying the truncation length adaptively. For RNNs with smooth activation functions, we prove that this bias controls the convergence rate of SGD with biased gradients for our non-convex loss. Using this theory, we develop a practical method for adaptively estimating the truncation length during training. We evaluate our adaptive TBPTT method on synthetic data and language modeling tasks and find that our adaptive TBPTT ameliorates the computational pitfalls of fixed TBPTT.","url_abs":"https://arxiv.org/abs/1905.07473v2","url_pdf":"https://arxiv.org/pdf/1905.07473v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptively-truncating-backpropagation-through","repo_url":"https://github.com/aicherc/adaptive_tbptt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1905.07473","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.07473"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aicherc/adaptive_tbptt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":21},"by_repo_kind":{"official":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1a697a4f5733df42","entry":"TPP_y_loss","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/temporal_pp_model.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/temporal_pp_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a697a4f5733df42"}},{"code_sha256_prefix":"f01e1ff3d3042d96","entry":"beta_est_method_to_params","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/synth_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/synth_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f01e1ff3d3042d96"}},{"code_sha256_prefix":"79a84db2ebe166bc","entry":"calculate_component_gradient_norms","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/adaptive_truncation.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/adaptive_truncation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79a84db2ebe166bc"}},{"code_sha256_prefix":"ed8aa9f6bc5882f9","entry":"calculate_elementwise_gradients","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/adaptive_truncation.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/adaptive_truncation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed8aa9f6bc5882f9"}},{"code_sha256_prefix":"180ab5fb456bcc2c","entry":"calculate_gradient_norms","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/adaptive_truncation.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/adaptive_truncation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"180ab5fb456bcc2c"}},{"code_sha256_prefix":"3e24b02cec0c8000","entry":"convert_dict_to_arg_string","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/script_maker.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/script_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e24b02cec0c8000"}},{"code_sha256_prefix":"df25a42af64b0733","entry":"generate_copy_sequence","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/generate_data.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df25a42af64b0733"}},{"code_sha256_prefix":"7e93cbc251f3eaed","entry":"generate_linear_seq","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/generate_data.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e93cbc251f3eaed"}},{"code_sha256_prefix":"b09f8ae9eca73a81","entry":"generate_repeating_sequence","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/generate_data.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b09f8ae9eca73a81"}},{"code_sha256_prefix":"488b1c7ff2dceb72","entry":"get_batch","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/language_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/language_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"488b1c7ff2dceb72"}},{"code_sha256_prefix":"b4a2dfd91c6a713c","entry":"get_batch","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/synth_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/synth_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4a2dfd91c6a713c"}},{"code_sha256_prefix":"8aa6056df6fad7cf","entry":"log_boxplot","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/plotting_utils.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/plotting_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aa6056df6fad7cf"}},{"code_sha256_prefix":"0e40a0fc6ff21389","entry":"log_heatmap","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/plotting_utils.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/plotting_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e40a0fc6ff21389"}},{"code_sha256_prefix":"385348b104576a5e","entry":"log_lineplot","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"tbptt/plotting_utils.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/tbptt/plotting_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"385348b104576a5e"}},{"code_sha256_prefix":"5ef6d0018f789dd0","entry":"process_data","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/temporal_pp_model.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/temporal_pp_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef6d0018f789dd0"}},{"code_sha256_prefix":"3ffead363144b94d","entry":"process_data_onehot","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/temporal_pp_model.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/temporal_pp_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ffead363144b94d"}},{"code_sha256_prefix":"73e9cbd44b3d4d2d","entry":"setup_copy_fixed_memory_script","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/script_maker.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/script_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73e9cbd44b3d4d2d"}},{"code_sha256_prefix":"7b3bab5b88fb01d2","entry":"setup_copy_variable_memory_script","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/script_maker.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/script_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b3bab5b88fb01d2"}},{"code_sha256_prefix":"5eacf6bc5dfa2fc4","entry":"stochastic_subset","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/language_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/language_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5eacf6bc5dfa2fc4"}},{"code_sha256_prefix":"9db61e1f7611e72d","entry":"stochastic_subset","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/synth_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/synth_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9db61e1f7611e72d"}},{"code_sha256_prefix":"b047609c78c58868","entry":"stochastic_subset","repo":"aicherc/adaptive_tbptt","repo_kind":"official","path":"experiments/temporal_pp_script.py","file_url":"https://github.com/aicherc/adaptive_tbptt/blob/HEAD/experiments/temporal_pp_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b047609c78c58868"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}