{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dimension-wise-importance-sampling-weight","title":"Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning","arxiv_id":"1905.02363","date":"2019-05-07","proceeding":null,"authors":["Seungyul Han","Youngchul Sung"],"abstract":"In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it difficult to reuse old samples with large IS weights. In this paper, we consider PPO, a representative on-policy algorithm, and propose its improvement by dimension-wise IS weight clipping which separately clips the IS weight of each action dimension to avoid large bias and adaptively controls the IS weight to bound policy update from the current policy. This new technique enables efficient learning for high action-dimensional tasks and reusing of old samples like in off-policy learning to increase the sample efficiency. Numerical results show that the proposed new algorithm outperforms PPO and other RL algorithms in various Open AI Gym tasks.","url_abs":"https://arxiv.org/abs/1905.02363v2","url_pdf":"https://arxiv.org/pdf/1905.02363v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dimension-wise-importance-sampling-weight","repo_url":"https://github.com/seungyulhan/disc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1905.02363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.02363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seungyulhan/disc","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":2,"ran_honours":1,"ran_violates":1,"unverified":13},"by_repo_kind":{"official":{"samples":19,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"8758308dc6c0727f","entry":"constfn","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/DISC/DISC.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/DISC/DISC.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8758308dc6c0727f"}},{"code_sha256_prefix":"98948fae5be7e92e","entry":"profile","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"98948fae5be7e92e"}},{"code_sha256_prefix":"a44574392d168384","entry":"register","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/models.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a44574392d168384"}},{"code_sha256_prefix":"a7b323241cd70612","entry":"safemean","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/DISC/DISC.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/DISC/DISC.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7b323241cd70612"}},{"code_sha256_prefix":"baf2a28d376d7803","entry":"sf01","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/DISC/DISC.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/DISC/DISC.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"baf2a28d376d7803"}},{"code_sha256_prefix":"aad5113d0b095522","entry":"shape_el","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/distributions.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"aad5113d0b095522"}},{"code_sha256_prefix":"a14fa85826d8091c","entry":"cg","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/cg.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/cg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a14fa85826d8091c"}},{"code_sha256_prefix":"2d39cd1166fa0004","entry":"colorize","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/console_util.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/console_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"2d39cd1166fa0004"}},{"code_sha256_prefix":"71bdb8bae1ed81f7","entry":"find_task_for_env_id_in_any_benchmark","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"71bdb8bae1ed81f7"}},{"code_sha256_prefix":"157f016aac49c483","entry":"fmt_item","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/console_util.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/console_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"157f016aac49c483"}},{"code_sha256_prefix":"d3ac76f7eab8f898","entry":"fmt_row","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/console_util.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/console_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d3ac76f7eab8f898"}},{"code_sha256_prefix":"7bd934aa47da390a","entry":"get_benchmark","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7bd934aa47da390a"}},{"code_sha256_prefix":"fa39f243af244088","entry":"get_monitor_files","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/bench/monitor.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/bench/monitor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fa39f243af244088"}},{"code_sha256_prefix":"3e82b9de30b1625f","entry":"get_task","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3e82b9de30b1625f"}},{"code_sha256_prefix":"404244405c7fd238","entry":"load_results","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/bench/monitor.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/bench/monitor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"404244405c7fd238"}},{"code_sha256_prefix":"b013c9fa08f46d68","entry":"make_atari","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/atari_wrappers.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/atari_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b013c9fa08f46d68"}},{"code_sha256_prefix":"b5272c7ba12ab1ee","entry":"make_output_format","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b5272c7ba12ab1ee"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/logger.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"671e38aae9f3b6ba","entry":"wrap_deepmind","repo":"seungyulhan/disc","repo_kind":"official","path":"baselines/common/atari_wrappers.py","file_url":"https://github.com/seungyulhan/disc/blob/HEAD/baselines/common/atari_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"671e38aae9f3b6ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}