{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/natural-actor-critic-for-robust-reinforcement","title":"Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation","arxiv_id":"2307.08875","date":"2023-07-17","proceeding":"NeurIPS 2023 11","authors":["Ruida Zhou","Tao Liu","Min Cheng","Dileep Kalathil","P. R. Kumar","Chao Tian"],"abstract":"We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.","url_abs":"https://arxiv.org/abs/2307.08875v2","url_pdf":"https://arxiv.org/pdf/2307.08875v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"natural-actor-critic-for-robust-reinforcement","repo_url":"https://github.com/tliu1997/rnac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.08875","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.08875"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tliu1997/RNAC","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tliu1997/rnac","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_honours":1,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"ae933dfddf8823fa","entry":"Actor_Beta","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae933dfddf8823fa"}},{"code_sha256_prefix":"b7e663349c62bb98","entry":"Actor_Gaussian","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7e663349c62bb98"}},{"code_sha256_prefix":"40af160a65b521e5","entry":"Critic","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40af160a65b521e5"}},{"code_sha256_prefix":"609d32bfa5631924","entry":"Normalization","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"609d32bfa5631924"}},{"code_sha256_prefix":"fe09cbc7d1ab50c5","entry":"PPO_continuous","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe09cbc7d1ab50c5"}},{"code_sha256_prefix":"31b7a842f737b97e","entry":"ReplayBuffer","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31b7a842f737b97e"}},{"code_sha256_prefix":"e7306d3fb1ac4b07","entry":"RewardScaling","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7306d3fb1ac4b07"}},{"code_sha256_prefix":"8b5734917fdb6af3","entry":"RunningMeanStd","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b5734917fdb6af3"}},{"code_sha256_prefix":"f74e37440bbe7726","entry":"evaluate_policy","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f74e37440bbe7726"}},{"code_sha256_prefix":"b49898d9bdfeb500","entry":"evaluate_policy","repo":"tliu1997/rnac","repo_kind":"official","path":"eval_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/eval_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b49898d9bdfeb500"}},{"code_sha256_prefix":"47b736f56e823740","entry":"load_agent","repo":"tliu1997/rnac","repo_kind":"official","path":"eval_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/eval_rnac.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47b736f56e823740"}},{"code_sha256_prefix":"787f96a4f64c5b9d","entry":"main","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"787f96a4f64c5b9d"}},{"code_sha256_prefix":"aaec1511480533d6","entry":"orthogonal_init","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aaec1511480533d6"}},{"code_sha256_prefix":"0b2b2d21541f3b92","entry":"save_agent","repo":"tliu1997/rnac","repo_kind":"official","path":"train_rnac.py","file_url":"https://github.com/tliu1997/rnac/blob/HEAD/train_rnac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b2b2d21541f3b92"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}