{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/svrg-for-policy-evaluation-with-fewer","title":"SVRG for Policy Evaluation with Fewer Gradient Evaluations","arxiv_id":"1906.03704","date":"2019-06-09","proceeding":null,"authors":["Zilun Peng","Ahmed Touati","Pascal Vincent","Doina Precup"],"abstract":"Stochastic variance-reduced gradient (SVRG) is an optimization method originally designed for tackling machine learning problems with a finite sum structure. SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy. SVRG makes use of gradient estimates at two scales. At the slower scale, SVRG computes a full gradient over the whole dataset, which could lead to prohibitive computation costs. In this work, we show that two variants of SVRG for policy evaluation could significantly diminish the number of gradient calculations while preserving a linear convergence speed. More importantly, our theoretical result implies that one does not need to use the entire dataset in every epoch of SVRG when it is applied to policy evaluation with linear function approximation. Our experiments demonstrate large computational savings provided by the proposed methods.","url_abs":"https://arxiv.org/abs/1906.03704v2","url_pdf":"https://arxiv.org/pdf/1906.03704v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"svrg-for-policy-evaluation-with-fewer","repo_url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.03704","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.03704"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cb5c7fe6e8682e01","entry":"calc_mspbe_torch","repo":"zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","repo_kind":"listed","path":"mspbe.py","file_url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients/blob/HEAD/mspbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb5c7fe6e8682e01"}},{"code_sha256_prefix":"3031d42074b687f2","entry":"get_pi_env","repo":"zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","repo_kind":"listed","path":"agent_env.py","file_url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients/blob/HEAD/agent_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3031d42074b687f2"}},{"code_sha256_prefix":"2767cc415ecf7032","entry":"mspbe_grad_omega","repo":"zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","repo_kind":"listed","path":"mspbe.py","file_url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients/blob/HEAD/mspbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2767cc415ecf7032"}},{"code_sha256_prefix":"d5eb28340e57ce2c","entry":"mspbe_grad_theta","repo":"zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients","repo_kind":"listed","path":"mspbe.py","file_url":"https://github.com/zilunpeng/svrg_for_policy_evaluation_with_fewer_gradients/blob/HEAD/mspbe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5eb28340e57ce2c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}