{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reward-scale-robustness-for-proximal-policy","title":"Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks","arxiv_id":"2310.17805","date":"2023-10-26","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Most reinforcement learning methods rely heavily on dense, well-normalized environment rewards. DreamerV3 recently introduced a model-based method with a number of tricks that mitigate these limitations, achieving state-of-the-art on a wide range of benchmarks with a single set of hyperparameters. This result sparked discussion about the generality of the tricks, since they appear to be applicable to other reinforcement learning algorithms. Our work applies DreamerV3's tricks to PPO and is the first such empirical study outside of the original work. Surprisingly, we find that the tricks presented do not transfer as general improvements to PPO. We use a high quality PPO reference implementation and present extensive ablation studies totaling over 10,000 A100 hours on the Arcade Learning Environment and the DeepMind Control Suite. Though our experiments demonstrate that these tricks do not generally outperform PPO, we identify cases where they succeed and offer insight into the relationship between the implementation tricks. In particular, PPO with these tricks performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping.","url_abs":"https://arxiv.org/abs/2310.17805v1","url_pdf":"https://arxiv.org/pdf/2310.17805v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.17805","atlas_url":"https://app.syntology.ai/?focus=2310.17805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17805"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/RyanNavillus/PPO-v3","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":3},"by_repo_kind":{"found_in_text":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"beb42d440ccf2925","entry":"layer_init","repo":"RyanNavillus/PPO-v3","repo_kind":"found_in_text","path":"ppo_v3/ppo_atari_envpool_resnet.py","file_url":"https://github.com/RyanNavillus/PPO-v3/blob/HEAD/ppo_v3/ppo_atari_envpool_resnet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"beb42d440ccf2925"}},{"code_sha256_prefix":"90d78613df850934","entry":"symexp","repo":"RyanNavillus/PPO-v3","repo_kind":"found_in_text","path":"ppo_v3/ppo_envpool_tricks.py","file_url":"https://github.com/RyanNavillus/PPO-v3/blob/HEAD/ppo_v3/ppo_envpool_tricks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90d78613df850934"}},{"code_sha256_prefix":"a6f6ca918a38c714","entry":"symexp","repo":"RyanNavillus/PPO-v3","repo_kind":"found_in_text","path":"ppo_v3/ppo_envpool_tricks_dmc.py","file_url":"https://github.com/RyanNavillus/PPO-v3/blob/HEAD/ppo_v3/ppo_envpool_tricks_dmc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6f6ca918a38c714"}},{"code_sha256_prefix":"2898f3c58f169917","entry":"symlog","repo":"RyanNavillus/PPO-v3","repo_kind":"found_in_text","path":"ppo_v3/ppo_envpool_tricks.py","file_url":"https://github.com/RyanNavillus/PPO-v3/blob/HEAD/ppo_v3/ppo_envpool_tricks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2898f3c58f169917"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}