{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/implicit-bias-of-policy-gradient-in-linear","title":"Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States","arxiv_id":"2402.07875","date":"2024-02-12","proceeding":null,"authors":["Noam Razin","Yotam Alexander","Edo Cohen-Karlik","Raja Giryes","Amir Globerson","Nadav Cohen"],"abstract":"In modern machine learning, models can often fit training data in numerous ways, some of which perform well on unseen (test) data, while others do not. Remarkably, in such cases gradient descent frequently exhibits an implicit bias that leads to excellent performance on unseen data. This implicit bias was extensively studied in supervised learning, but is far less understood in optimal control (reinforcement learning). There, learning a controller applied to a system via gradient descent is known as policy gradient, and a question of prime importance is the extent to which a learned controller extrapolates to unseen initial states. This paper theoretically studies the implicit bias of policy gradient in terms of extrapolation to unseen initial states. Focusing on the fundamental Linear Quadratic Regulator (LQR) problem, we establish that the extent of extrapolation depends on the degree of exploration induced by the system when commencing from initial states included in training. Experiments corroborate our theory, and demonstrate its conclusions on problems beyond LQR, where systems are non-linear and controllers are neural networks. We hypothesize that real-world optimal control may be greatly improved by developing methods for informed selection of initial states to train on.","url_abs":"https://arxiv.org/abs/2402.07875v2","url_pdf":"https://arxiv.org/pdf/2402.07875v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"implicit-bias-of-policy-gradient-in-linear","repo_url":"https://github.com/noamrazin/imp_bias_control","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.07875","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07875"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/noamrazin/imp_bias_control","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"0516975f7458a910","entry":"create_custom_initial_states","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/experiment/pendulum_co_experiment.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/experiment/pendulum_co_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0516975f7458a910"}},{"code_sha256_prefix":"50e446097c1570a2","entry":"euler_matrix","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/models/quadcopter.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/models/quadcopter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50e446097c1570a2"}},{"code_sha256_prefix":"798206d9fab33fab","entry":"create_custom_random_linear_system","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/experiment/lqr_co_experiment.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/experiment/lqr_co_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"798206d9fab33fab"}},{"code_sha256_prefix":"3ff77506ae5c6eb5","entry":"create_identity_linear_system","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/experiment/lqr_co_experiment.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/experiment/lqr_co_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ff77506ae5c6eb5"}},{"code_sha256_prefix":"c7d18e373228b790","entry":"create_pendulum_dynamics_animation","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/models/pendulum.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/models/pendulum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7d18e373228b790"}},{"code_sha256_prefix":"17c7da9d10d73fe9","entry":"create_shift_linear_system","repo":"noamrazin/imp_bias_control","repo_kind":"official","path":"control/experiment/lqr_co_experiment.py","file_url":"https://github.com/noamrazin/imp_bias_control/blob/HEAD/control/experiment/lqr_co_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17c7da9d10d73fe9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}