{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pc-pg-policy-cover-directed-exploration-for","title":"PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning","arxiv_id":"2007.08459","date":"2020-07-16","proceeding":"NeurIPS 2020 12","authors":["Alekh Agarwal","Mikael Henaff","Sham Kakade","Wen Sun"],"abstract":"Direct policy gradient methods for reinforcement learning are a successful approach for a variety of reasons: they are model free, they directly optimize the performance metric of interest, and they allow for richly parameterized policies. Their primary drawback is that, by being local in nature, they fail to adequately explore the environment. In contrast, while model-based approaches and Q-learning directly handle exploration through the use of optimism, their ability to handle model misspecification and function approximation is far less evident. This work introduces the the Policy Cover-Policy Gradient (PC-PG) algorithm, which provably balances the exploration vs. exploitation tradeoff using an ensemble of learned policies (the policy cover). PC-PG enjoys polynomial sample complexity and run time for both tabular MDPs and, more generally, linear MDPs in an infinite dimensional RKHS. Furthermore, PC-PG also has strong guarantees under model misspecification that go beyond the standard worst case $\\ell_{\\infty}$ assumptions; this includes approximation guarantees for state aggregation under an average case error assumption, along with guarantees under a more general assumption where the approximation error under distribution shift is controlled. We complement the theory with empirical evaluation across a variety of domains in both reward-free and reward-driven settings.","url_abs":"https://arxiv.org/abs/2007.08459v2","url_pdf":"https://arxiv.org/pdf/2007.08459v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pc-pg-policy-cover-directed-exploration-for","repo_url":"https://github.com/mbhenaff/pcpg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"q-learning","task_name":"Q-Learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.08459","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.08459"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mbhenaff/pcpg","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"listed":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"0173e4280d9a9884","entry":"flatten_lists","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0173e4280d9a9884"}},{"code_sha256_prefix":"bcd8b4acab199405","entry":"make_output_format","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcd8b4acab199405"}},{"code_sha256_prefix":"a44574392d168384","entry":"register","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/common/models.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a44574392d168384"}},{"code_sha256_prefix":"2bf5c781f8e4aad5","entry":"build_impala_cnn","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/common/models.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bf5c781f8e4aad5"}},{"code_sha256_prefix":"91a934c3d796a9d4","entry":"profile","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91a934c3d796a9d4"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"4ab4ad5ff9820e39","entry":"ts2xy","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ab4ad5ff9820e39"}},{"code_sha256_prefix":"43741b9339bd80da","entry":"window_func","repo":"mbhenaff/pcpg","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/mbhenaff/pcpg/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43741b9339bd80da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}