{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trajectory-aware-eligibility-traces-for-off","title":"Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning","arxiv_id":"2301.11321","date":"2023-01-26","proceeding":null,"authors":["Brett Daley","Martha White","Christopher Amato","Marlos C. Machado"],"abstract":"Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning, but counteracting off-policy bias without exacerbating variance is challenging. Classically, off-policy bias is corrected in a per-decision manner: past temporal-difference errors are re-weighted by the instantaneous Importance Sampling (IS) ratio after each action via eligibility traces. Many off-policy algorithms rely on this mechanism, along with differing protocols for cutting the IS ratios to combat the variance of the IS estimator. Unfortunately, once a trace has been fully cut, the effect cannot be reversed. This has led to the development of credit-assignment strategies that account for multiple past experiences at a time. These trajectory-aware methods have not been extensively analyzed, and their theoretical justification remains uncertain. In this paper, we propose a multistep operator that can express both per-decision and trajectory-aware methods. We prove convergence conditions for our operator in the tabular setting, establishing the first guarantees for several existing methods as well as many new ones. Finally, we introduce Recency-Bounded Importance Sampling (RBIS), which leverages trajectory awareness to perform robustly across $\\lambda$-values in an off-policy control task.","url_abs":"https://arxiv.org/abs/2301.11321v2","url_pdf":"https://arxiv.org/pdf/2301.11321v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trajectory-aware-eligibility-traces-for-off","repo_url":"https://github.com/brett-daley/trajectory-aware-etraces","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.11321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.11321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brett-daley/trajectory-aware-etraces","reach":null}],"summary":{"ran":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e14bff5f93f63a9c","entry":"RBIS","repo":"brett-daley/trajectory-aware-etraces","repo_kind":"official","path":"trajectory_aware_etraces/algorithms/algorithms.py","file_url":"https://github.com/brett-daley/trajectory-aware-etraces/blob/HEAD/trajectory_aware_etraces/algorithms/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e14bff5f93f63a9c"}},{"code_sha256_prefix":"5136498ee8aebc76","entry":"Vector","repo":"brett-daley/trajectory-aware-etraces","repo_kind":"official","path":"trajectory_aware_etraces/algorithms/algorithms.py","file_url":"https://github.com/brett-daley/trajectory-aware-etraces/blob/HEAD/trajectory_aware_etraces/algorithms/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5136498ee8aebc76"}},{"code_sha256_prefix":"6addba60737c1bdf","entry":"reduce_sum","repo":"brett-daley/trajectory-aware-etraces","repo_kind":"official","path":"trajectory_aware_etraces/algorithms/algorithms.py","file_url":"https://github.com/brett-daley/trajectory-aware-etraces/blob/HEAD/trajectory_aware_etraces/algorithms/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6addba60737c1bdf"}},{"code_sha256_prefix":"cb3a59ff9f469385","entry":"EligibilityTraces","repo":"brett-daley/trajectory-aware-etraces","repo_kind":"official","path":"trajectory_aware_etraces/algorithms/algorithms.py","file_url":"https://github.com/brett-daley/trajectory-aware-etraces/blob/HEAD/trajectory_aware_etraces/algorithms/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb3a59ff9f469385"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}