{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sample-dropout-a-simple-yet-effective","title":"Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization","arxiv_id":"2302.02299","date":"2023-02-05","proceeding":null,"authors":["Zichuan Lin","Xiapeng Wu","Mingfei Sun","Deheng Ye","Qiang Fu","Wei Yang","Wei Liu"],"abstract":"Recent success in Deep Reinforcement Learning (DRL) methods has shown that policy optimization with respect to an off-policy distribution via importance sampling is effective for sample reuse. In this paper, we show that the use of importance sampling could introduce high variance in the objective estimate. Specifically, we show in a principled way that the variance of importance sampling estimate grows quadratically with importance ratios and the large ratios could consequently jeopardize the effectiveness of surrogate objective optimization. We then propose a technique called sample dropout to bound the estimation variance by dropping out samples when their ratio deviation is too high. We instantiate this sample dropout technique on representative policy optimization algorithms, including TRPO, PPO, and ESPO, and demonstrate that it consistently boosts the performance of those DRL algorithms on both continuous and discrete action controls, including MuJoCo, DMControl and Atari video games. Our code is open-sourced at \\url{https://github.com/LinZichuan/sdpo.git}.","url_abs":"https://arxiv.org/abs/2302.02299v1","url_pdf":"https://arxiv.org/pdf/2302.02299v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sample-dropout-a-simple-yet-effective","repo_url":"https://github.com/linzichuan/sdpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"mujoco","task_name":"MuJoCo"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"trpo","method_name":"TRPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.02299","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.02299"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linzichuan/sdpo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"f29202d8108bccd3","entry":"flat_concat","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/ppo2/model.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/ppo2/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f29202d8108bccd3"}},{"code_sha256_prefix":"bcd8b4acab199405","entry":"make_output_format","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"bcd8b4acab199405"}},{"code_sha256_prefix":"a44574392d168384","entry":"register","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/common/models.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a44574392d168384"}},{"code_sha256_prefix":"2bf5c781f8e4aad5","entry":"build_impala_cnn","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/common/models.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"2bf5c781f8e4aad5"}},{"code_sha256_prefix":"a3b4038f4e9b9479","entry":"cat_entropy","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a3b4038f4e9b9479"}},{"code_sha256_prefix":"7082b4bbc0d9384f","entry":"cat_entropy_softmax","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7082b4bbc0d9384f"}},{"code_sha256_prefix":"91a934c3d796a9d4","entry":"profile","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"91a934c3d796a9d4"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"178099b1f8fb2b9d","entry":"sample","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/a2c/utils.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"178099b1f8fb2b9d"}},{"code_sha256_prefix":"170bf6ea6525be34","entry":"ts2xy_monitor","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"170bf6ea6525be34"}},{"code_sha256_prefix":"43741b9339bd80da","entry":"window_func","repo":"linzichuan/sdpo","repo_kind":"official","path":"baselines/results_plotter.py","file_url":"https://github.com/linzichuan/sdpo/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"43741b9339bd80da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}