{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/global-convergence-of-receding-horizon-policy","title":"Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designs","arxiv_id":"2309.04831","date":"2023-09-09","proceeding":null,"authors":["Xiangyuan Zhang","Saviz Mowlavi","Mouhacine Benosman","Tamer Başar"],"abstract":"We introduce the receding-horizon policy gradient (RHPG) algorithm, the first PG algorithm with provable global convergence in learning the optimal linear estimator designs, i.e., the Kalman filter (KF). Notably, the RHPG algorithm does not require any prior knowledge of the system for initialization and does not require the target system to be open-loop stable. The key of RHPG is that we integrate vanilla PG (or any other policy search directions) into a dynamic programming outer loop, which iteratively decomposes the infinite-horizon KF problem that is constrained and non-convex in the policy parameter into a sequence of static estimation problems that are unconstrained and strongly-convex, thus enabling global convergence. We further provide fine-grained analyses of the optimization landscape under RHPG and detail the convergence and sample complexity guarantees of the algorithm. This work serves as an initial attempt to develop reinforcement learning algorithms specifically for control applications with performance guarantees by utilizing classic control theory in both algorithmic design and theoretical analyses. Lastly, we validate our theories by deploying the RHPG algorithm to learn the Kalman filter design of a large-scale convection-diffusion model. We open-source the code repository at \\url{https://github.com/xiangyuan-zhang/LearningKF}.","url_abs":"https://arxiv.org/abs/2309.04831v1","url_pdf":"https://arxiv.org/pdf/2309.04831v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"global-convergence-of-receding-horizon-policy","repo_url":"https://github.com/xiangyuan-zhang/learningkf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.04831","atlas_url":"https://app.syntology.ai/?focus=2309.04831","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.04831"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiangyuan-zhang/learningkf","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"72ee124a82668991","entry":"finite_kf","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"utils/mb_control.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/utils/mb_control.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72ee124a82668991"}},{"code_sha256_prefix":"0e79ac969862e77f","entry":"ft_matrix","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/utils/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e79ac969862e77f"}},{"code_sha256_prefix":"f2d64054541ec016","entry":"infinite_kf","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"utils/mb_control.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/utils/mb_control.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2d64054541ec016"}},{"code_sha256_prefix":"5b61c027ffb39641","entry":"pde_setup","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/utils/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b61c027ffb39641"}},{"code_sha256_prefix":"847fa94865852456","entry":"s_radius","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/utils/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"847fa94865852456"}},{"code_sha256_prefix":"c3918f99076b5533","entry":"adam_opt","repo":"xiangyuan-zhang/learningkf","repo_kind":"official","path":"algorithm/rhpg_kf.py","file_url":"https://github.com/xiangyuan-zhang/learningkf/blob/HEAD/algorithm/rhpg_kf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3918f99076b5533"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}