{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/target-networks-and-over-parameterization","title":"Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation","arxiv_id":"2405.21043","date":"2024-05-31","proceeding":null,"authors":["Fengdi Che","Chenjun Xiao","Jincheng Mei","Bo Dai","Ramki Gummadi","Oscar A Ramirez","Christopher K Harris","A. Rupam Mahmood","Dale Schuurmans"],"abstract":"We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data. Our condition is naturally satisfied for expected updates over the entire state-action space or learning with a batch of complete trajectories from episodic Markov decision processes. Notably, using only a target network or an over-parameterized model does not provide such a convergence guarantee. Additionally, we extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications, akin to value truncation for the final states in trajectories. Our primary result focuses on temporal difference estimation for prediction, providing high-probability value estimation error bounds and empirical analysis on Baird's counterexample and a Four-room task. Furthermore, we explore the control setting, demonstrating that similar convergence conditions apply to Q-learning.","url_abs":"https://arxiv.org/abs/2405.21043v2","url_pdf":"https://arxiv.org/pdf/2405.21043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"target-networks-and-over-parameterization","repo_url":"https://github.com/FengdiC/OTTD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"q-learning","task_name":"Q-Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.21043","atlas_url":"https://app.syntology.ai/?focus=2405.21043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.21043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FengdiC/OTTD","reach":null}],"summary":{"ran_honours":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e06fa0891d5eb273","entry":"RM","repo":"FengdiC/OTTD","repo_kind":"official","path":"baird.py","file_url":"https://github.com/FengdiC/OTTD/blob/HEAD/baird.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e06fa0891d5eb273"}},{"code_sha256_prefix":"151d6fd7e4bff9c1","entry":"TD","repo":"FengdiC/OTTD","repo_kind":"official","path":"baird.py","file_url":"https://github.com/FengdiC/OTTD/blob/HEAD/baird.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"151d6fd7e4bff9c1"}},{"code_sha256_prefix":"388227820d38574c","entry":"TD_target","repo":"FengdiC/OTTD","repo_kind":"official","path":"baird.py","file_url":"https://github.com/FengdiC/OTTD/blob/HEAD/baird.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"388227820d38574c"}},{"code_sha256_prefix":"0133639600c81d45","entry":"compute_IS","repo":"FengdiC/OTTD","repo_kind":"official","path":"learn_four_room.py","file_url":"https://github.com/FengdiC/OTTD/blob/HEAD/learn_four_room.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0133639600c81d45"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}