{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hindsight-policy-gradients","title":"Hindsight policy gradients","arxiv_id":"1711.06006","date":"2017-11-16","proceeding":"ICLR 2019 5","authors":["Paulo Rauber","Avinash Ummadisingu","Filipe Mutz","Juergen Schmidhuber"],"abstract":"A reinforcement learning agent that needs to pursue different goals across\nepisodes requires a goal-conditional policy. In addition to their potential to\ngeneralize desirable behavior to unseen goals, such policies may also enable\nhigher-level planning based on subgoals. In sparse-reward environments, the\ncapacity to exploit information about the degree to which an arbitrary goal has\nbeen achieved while another goal was intended appears crucial to enable sample\nefficient learning. However, reinforcement learning agents have only recently\nbeen endowed with such capacity for hindsight. In this paper, we demonstrate\nhow hindsight can be introduced to policy gradient methods, generalizing this\nidea to a broad class of successful algorithms. Our experiments on a diverse\nselection of sparse-reward environments show that hindsight leads to a\nremarkable increase in sample efficiency.","url_abs":"http://arxiv.org/abs/1711.06006v3","url_pdf":"http://arxiv.org/pdf/1711.06006v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hindsight-policy-gradients","repo_url":"https://github.com/paulorauber/hpg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1711.06006","atlas_url":"https://app.syntology.ai/?focus=1711.06006","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1711.06006"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/paulorauber/hpg","reach":null}],"summary":{"ran_honours":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"625d10a8356984c1","entry":"average_performance","repo":"paulorauber/hpg","repo_kind":"listed","path":"hpg/scripts/analysis.py","file_url":"https://github.com/paulorauber/hpg/blob/HEAD/hpg/scripts/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"625d10a8356984c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}