{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystifying-the-recency-heuristic-in","title":"Demystifying the Recency Heuristic in Temporal-Difference Learning","arxiv_id":"2406.12284","date":"2024-06-18","proceeding":null,"authors":["Brett Daley","Marlos C. Machado","Martha White"],"abstract":"The recency heuristic in reinforcement learning is the assumption that stimuli that occurred closer in time to an acquired reward should be more heavily reinforced. The recency heuristic is one of the key assumptions made by TD($\\lambda$), which reinforces recent experiences according to an exponentially decaying weighting. In fact, all other widely used return estimators for TD learning, such as $n$-step returns, satisfy a weaker (i.e., non-monotonic) recency heuristic. Why is the recency heuristic effective for temporal credit assignment? What happens when credit is assigned in a way that violates this heuristic? In this paper, we analyze the specific mathematical implications of adopting the recency heuristic in TD learning. We prove that any return estimator satisfying this heuristic: 1) is guaranteed to converge to the correct value function, 2) has a relatively fast contraction rate, and 3) has a long window of effective credit assignment, yet bounded worst-case variance. We also give a counterexample where on-policy, tabular TD methods violating the recency heuristic diverge. Our results offer some of the first theoretical evidence that credit assignment based on the recency heuristic facilitates learning.","url_abs":"https://arxiv.org/abs/2406.12284v2","url_pdf":"https://arxiv.org/pdf/2406.12284v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystifying-the-recency-heuristic-in","repo_url":"https://github.com/brett-daley/recency-heuristic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.12284","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12284"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brett-daley/recency-heuristic","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"12cca2249119d079","entry":"calculate","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"learning/return_estimators.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/learning/return_estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12cca2249119d079"}},{"code_sha256_prefix":"dcf8fb5c2dde194d","entry":"calculate_auc","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"plotting/utils.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/plotting/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcf8fb5c2dde194d"}},{"code_sha256_prefix":"22fa413b90a3f7ce","entry":"get_sparse_lambda_func","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"plotting/plot_eligibility.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/plotting/plot_eligibility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22fa413b90a3f7ce"}},{"code_sha256_prefix":"3bf188221615e214","entry":"get_trunc_lambda_func","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"plotting/plot_eligibility.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/plotting/plot_eligibility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bf188221615e214"}},{"code_sha256_prefix":"363d1c954acd4477","entry":"parse_extra","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"cmdline/parse.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/cmdline/parse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"363d1c954acd4477"}},{"code_sha256_prefix":"bde5baf8af452a92","entry":"parse_kwargs","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"cmdline/parse.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/cmdline/parse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bde5baf8af452a92"}},{"code_sha256_prefix":"202dd22e01f0f721","entry":"policy_evaluation","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"learning/dynamic_programming.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/learning/dynamic_programming.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"202dd22e01f0f721"}},{"code_sha256_prefix":"2b0b94149a0318a6","entry":"try_cast","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"cmdline/parse.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/cmdline/parse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b0b94149a0318a6"}},{"code_sha256_prefix":"5a2d82c50fce3111","entry":"insert_defaults","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"cmdline/defaults.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/cmdline/defaults.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a2d82c50fce3111"}},{"code_sha256_prefix":"8d4d29fcc36bc1b4","entry":"load_defaults","repo":"brett-daley/recency-heuristic","repo_kind":"official","path":"cmdline/defaults.py","file_url":"https://github.com/brett-daley/recency-heuristic/blob/HEAD/cmdline/defaults.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d4d29fcc36bc1b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}