{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuous-time-fitted-value-iteration-for","title":"Continuous-Time Fitted Value Iteration for Robust Policies","arxiv_id":"2110.01954","date":"2021-10-05","proceeding":null,"authors":["Michael Lutter","Boris Belousov","Shie Mannor","Dieter Fox","Animesh Garg","Jan Peters"],"abstract":"Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. Especially for continuous control, solving this differential equation and its extension the Hamilton-Jacobi-Isaacs equation, is important as it yields the optimal policy that achieves the maximum reward on a give task. In the case of the Hamilton-Jacobi-Isaacs equation, which includes an adversary controlling the environment and minimizing the reward, the obtained policy is also robust to perturbations of the dynamics. In this paper we propose continuous fitted value iteration (cFVI) and robust fitted value iteration (rFVI). These algorithms leverage the non-linear control-affine dynamics and separable state and action reward of many continuous control problems to derive the optimal policy and optimal adversary in closed form. This analytic expression simplifies the differential equations and enables us to solve for the optimal value function using value iteration for continuous actions and states as well as the adversarial case. Notably, the resulting algorithms do not require discretization of states or actions. We apply the resulting algorithms to the Furuta pendulum and cartpole. We show that both algorithms obtain the optimal policy. The robustness Sim2Real experiments on the physical systems show that the policies successfully achieve the task in the real-world. When changing the masses of the pendulum, we observe that robust value iteration is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm. Videos of the experiments are shown at https://sites.google.com/view/rfvi","url_abs":"https://arxiv.org/abs/2110.01954v1","url_pdf":"https://arxiv.org/pdf/2110.01954v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuous-time-fitted-value-iteration-for","repo_url":"https://github.com/milutter/value_iteration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.01954","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.01954"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/milutter/value_iteration","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"listed":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"899cccacb2a8110b","entry":"analyze_error","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/utils.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"899cccacb2a8110b"}},{"code_sha256_prefix":"a1fcede1d2566d1f","entry":"bounds","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/update_value_function.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/update_value_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1fcede1d2566d1f"}},{"code_sha256_prefix":"a15d983a455190b6","entry":"convex_conjugate_test","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/cost_functions/convex_conjugate_function.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/cost_functions/convex_conjugate_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a15d983a455190b6"}},{"code_sha256_prefix":"b1292700140eb893","entry":"error_statistics_string","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/utils.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1292700140eb893"}},{"code_sha256_prefix":"a1d1c9c9a8a751af","entry":"evaluate","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/utils.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d1c9c9a8a751af"}},{"code_sha256_prefix":"10793fdeb1bd8782","entry":"norm","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/update_value_function.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/update_value_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10793fdeb1bd8782"}},{"code_sha256_prefix":"7b99030f39e97e88","entry":"sigmoid","repo":"milutter/value_iteration","repo_kind":"listed","path":"value_iteration/cost_functions/utils.py","file_url":"https://github.com/milutter/value_iteration/blob/HEAD/value_iteration/cost_functions/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b99030f39e97e88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}