{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-hybrid-stochastic-policy-gradient-algorithm","title":"A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning","arxiv_id":"2003.00430","date":"2020-03-01","proceeding":null,"authors":["Nhan H. Pham","Lam M. Nguyen","Dzung T. Phan","Phuong Ha Nguyen","Marten van Dijk","Quoc Tran-Dinh"],"abstract":"We propose a novel hybrid stochastic policy gradient estimator by combining an unbiased policy gradient estimator, the REINFORCE estimator, with another biased one, an adapted SARAH estimator for policy optimization. The hybrid policy gradient estimator is shown to be biased, but has variance reduced property. Using this estimator, we develop a new Proximal Hybrid Stochastic Policy Gradient Algorithm (ProxHSPGA) to solve a composite policy optimization problem that allows us to handle constraints or regularizers on the policy parameters. We first propose a single-looped algorithm then introduce a more practical restarting variant. We prove that both algorithms can achieve the best-known trajectory complexity $\\mathcal{O}\\left(\\varepsilon^{-3}\\right)$ to attain a first-order stationary point for the composite problem which is better than existing REINFORCE/GPOMDP $\\mathcal{O}\\left(\\varepsilon^{-4}\\right)$ and SVRPG $\\mathcal{O}\\left(\\varepsilon^{-10/3}\\right)$ in the non-composite setting. We evaluate the performance of our algorithm on several well-known examples in reinforcement learning. Numerical results show that our algorithm outperforms two existing methods on these examples. Moreover, the composite settings indeed have some advantages compared to the non-composite ones on certain problems.","url_abs":"https://arxiv.org/abs/2003.00430v2","url_pdf":"https://arxiv.org/pdf/2003.00430v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-hybrid-stochastic-policy-gradient-algorithm","repo_url":"https://github.com/unc-optimization/ProxHSPGA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"reinforce","method_name":"REINFORCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2003.00430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.00430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unc-optimization/ProxHSPGA","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"78f3b67c1b7a49ae","entry":"extract_path","repo":"unc-optimization/ProxHSPGA","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/unc-optimization/ProxHSPGA/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78f3b67c1b7a49ae"}},{"code_sha256_prefix":"d818381a9f679b96","entry":"prox_l1_norm","repo":"unc-optimization/ProxHSPGA","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/unc-optimization/ProxHSPGA/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d818381a9f679b96"}},{"code_sha256_prefix":"23f17d1d35ce1ccf","entry":"prox_l2_square","repo":"unc-optimization/ProxHSPGA","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/unc-optimization/ProxHSPGA/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23f17d1d35ce1ccf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}