{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/policy-optimization-through-approximated","title":"Policy Optimization Through Approximate Importance Sampling","arxiv_id":"1910.03857","date":"2019-10-09","proceeding":null,"authors":["Marcin B. Tomczak","Dongho Kim","Peter Vrancx","Kee-Eung Kim"],"abstract":"Recent policy optimization approaches (Schulman et al., 2015a; 2017) have achieved substantial empirical successes by constructing new proxy optimization objectives. These proxy objectives allow stable and low variance policy learning, but require small policy updates to ensure that the proxy objective remains an accurate approximation of the target policy value. In this paper we derive an alternative objective that obtains the value of the target policy by applying importance sampling (IS). However, the basic importance sampled objective is not suitable for policy optimization, as it incurs too high variance in policy updates. We therefore introduce an approximation that allows us to directly trade-off the bias of approximation with the variance in policy updates. We show that our approximation unifies previously developed approaches and allows us to interpolate between them. We develop a practical algorithm by optimizing the introduced objective with proximal policy optimization techniques (Schulman et al., 2017). We also provide a theoretical analysis of the introduced policy optimization objective demonstrating bias-variance trade-off. We empirically demonstrate that the resulting algorithm improves upon state of the art on-policy policy optimization on continuous control benchmarks.","url_abs":"https://arxiv.org/abs/1910.03857v2","url_pdf":"https://arxiv.org/pdf/1910.03857v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"policy-optimization-through-approximated","repo_url":"https://github.com/marctom/POTAIS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.03857","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.03857"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marctom/POTAIS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0173e4280d9a9884","entry":"flatten_lists","repo":"marctom/POTAIS","repo_kind":"official","path":"algorithm.py","file_url":"https://github.com/marctom/POTAIS/blob/HEAD/algorithm.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0173e4280d9a9884"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}