{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-learning-intrinsic-rewards-for-policy","title":"On Learning Intrinsic Rewards for Policy Gradient Methods","arxiv_id":"1804.06459","date":"2018-04-17","proceeding":"NeurIPS 2018 12","authors":["Zeyu Zheng","Junhyuk Oh","Satinder Singh"],"abstract":"In many sequential decision making tasks, it is challenging to design reward\nfunctions that help an RL agent efficiently learn behavior that is considered\ngood by the agent designer. A number of different formulations of the\nreward-design problem, or close variants thereof, have been proposed in the\nliterature. In this paper we build on the Optimal Rewards Framework of Singh\net.al. that defines the optimal intrinsic reward function as one that when used\nby an RL agent achieves behavior that optimizes the task-specifying or\nextrinsic reward function. Previous work in this framework has shown how good\nintrinsic reward functions can be learned for lookahead search based planning\nagents. Whether it is possible to learn intrinsic reward functions for learning\nagents remains an open problem. In this paper we derive a novel algorithm for\nlearning intrinsic rewards for policy-gradient based learning agents. We\ncompare the performance of an augmented agent that uses our algorithm to\nprovide additive intrinsic rewards to an A2C-based policy learner (for Atari\ngames) and a PPO-based policy learner (for Mujoco domains) with a baseline\nagent that uses the same policy learners but with only extrinsic rewards. Our\nresults show improved performance on most but not all of the domains.","url_abs":"http://arxiv.org/abs/1804.06459v2","url_pdf":"http://arxiv.org/pdf/1804.06459v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-learning-intrinsic-rewards-for-policy","repo_url":"https://github.com/Hwhitetooth/lirpg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1804.06459","atlas_url":"https://app.syntology.ai/?focus=1804.06459","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.06459"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Hwhitetooth/lirpg","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_honours":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"98948fae5be7e92e","entry":"profile","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"98948fae5be7e92e"}},{"code_sha256_prefix":"a7b323241cd70612","entry":"safemean","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/a2c/a2c.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/a2c/a2c.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a7b323241cd70612"}},{"code_sha256_prefix":"a3b4038f4e9b9479","entry":"cat_entropy","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a3b4038f4e9b9479"}},{"code_sha256_prefix":"7082b4bbc0d9384f","entry":"cat_entropy_softmax","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7082b4bbc0d9384f"}},{"code_sha256_prefix":"5eee517402b390d5","entry":"make_output_format","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5eee517402b390d5"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"178099b1f8fb2b9d","entry":"sample","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"178099b1f8fb2b9d"}},{"code_sha256_prefix":"8c5b970d902942f8","entry":"ts2xy","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5b970d902942f8"}},{"code_sha256_prefix":"43741b9339bd80da","entry":"window_func","repo":"Hwhitetooth/lirpg","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/Hwhitetooth/lirpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"43741b9339bd80da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}