{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reinforcement-learning-with-perturbed-rewards","title":"Reinforcement Learning with Perturbed Rewards","arxiv_id":"1810.01032","date":"2018-10-02","proceeding":"ICLR 2019 5","authors":["Jingkang Wang","Yang Liu","Bo Li"],"abstract":"Recent studies have shown that reinforcement learning (RL) models are vulnerable in various noisy scenarios. For instance, the observed reward channel is often subject to noise in practice (e.g., when rewards are collected through sensors), and is therefore not credible. In addition, for applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors by receiving corrupted rewards. In this paper, we consider noisy RL problems with perturbed rewards, which can be approximated with a confusion matrix. We develop a robust RL framework that enables agents to learn in noisy environments where only perturbed rewards are observed. Our solution framework builds on existing RL/DRL algorithms and firstly addresses the biased noisy reward setting without any assumptions on the true distribution (e.g., zero-mean Gaussian noise as made in previous works). The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that trained policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 84.6% and 80.8% improvements on average score for five Atari games, with error rates as 10% and 30% respectively.","url_abs":"https://arxiv.org/abs/1810.01032v4","url_pdf":"https://arxiv.org/pdf/1810.01032v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reinforcement-learning-with-perturbed-rewards","repo_url":"https://github.com/wangjksjtu/rl-perturbed-reward","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.01032","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.01032"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangjksjtu/rl-perturbed-reward","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4638501194fd0111","entry":"build_state","repo":"wangjksjtu/rl-perturbed-reward","repo_kind":"official","path":"gym-control/noise_estimator.py","file_url":"https://github.com/wangjksjtu/rl-perturbed-reward/blob/HEAD/gym-control/noise_estimator.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4638501194fd0111"}},{"code_sha256_prefix":"a0e65d9285742b87","entry":"to_bin","repo":"wangjksjtu/rl-perturbed-reward","repo_kind":"official","path":"gym-control/noise_estimator.py","file_url":"https://github.com/wangjksjtu/rl-perturbed-reward/blob/HEAD/gym-control/noise_estimator.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0e65d9285742b87"}},{"code_sha256_prefix":"550f10e6803fc922","entry":"initialize_cmat","repo":"wangjksjtu/rl-perturbed-reward","repo_kind":"official","path":"gym-atari/baselines/baselines/noisy_reward.py","file_url":"https://github.com/wangjksjtu/rl-perturbed-reward/blob/HEAD/gym-atari/baselines/baselines/noisy_reward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"550f10e6803fc922"}},{"code_sha256_prefix":"06edab9d1133a8dc","entry":"is_invertible","repo":"wangjksjtu/rl-perturbed-reward","repo_kind":"official","path":"gym-atari/baselines/baselines/noisy_reward.py","file_url":"https://github.com/wangjksjtu/rl-perturbed-reward/blob/HEAD/gym-atari/baselines/baselines/noisy_reward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06edab9d1133a8dc"}},{"code_sha256_prefix":"414baf4e279c0209","entry":"smooth","repo":"wangjksjtu/rl-perturbed-reward","repo_kind":"official","path":"gym-control/plot.py","file_url":"https://github.com/wangjksjtu/rl-perturbed-reward/blob/HEAD/gym-control/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"414baf4e279c0209"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}