{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-closer-look-at-invalid-action-masking-in","title":"A Closer Look at Invalid Action Masking in Policy Gradient Algorithms","arxiv_id":"2006.14171","date":"2020-06-25","proceeding":null,"authors":["Shengyi Huang","Santiago Ontañón"],"abstract":"In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved state-of-the-art performance in many challenging strategy games. Because these games have complicated rules, an action sampled from the full discrete action distribution predicted by the learned policy is likely to be invalid according to the game rules (e.g., walking into a wall). The usual approach to deal with this problem in policy gradient algorithms is to \"mask out\" invalid actions and just sample from the set of valid actions. The implications of this process, however, remain under-investigated. In this paper, we 1) show theoretical justification for such a practice, 2) empirically demonstrate its importance as the space of invalid actions grows, and 3) provide further insights by evaluating different action masking regimes, such as removing masking after an agent has been trained using masking. The source code can be found at https://github.com/vwxyzjn/invalid-action-masking","url_abs":"https://arxiv.org/abs/2006.14171v3","url_pdf":"https://arxiv.org/pdf/2006.14171v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-closer-look-at-invalid-action-masking-in","repo_url":"https://github.com/vwxyzjn/invalid-action-masking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-closer-look-at-invalid-action-masking-in","repo_url":"https://github.com/Stable-Baselines-Team/stable-baselines3-contrib","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"real-time-strategy-games","task_name":"Real-Time Strategy Games"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.14171","atlas_url":"https://app.syntology.ai/?focus=2006.14171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.14171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Stable-Baselines-Team/stable-baselines3-contrib","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vwxyzjn/invalid-action-masking","reach":null}],"summary":{"ran":3,"ran_fixture":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"b720173db270f08b","entry":"conjugate_gradient_solver","repo":"Stable-Baselines-Team/stable-baselines3-contrib","repo_kind":"listed","path":"sb3_contrib/common/utils.py","file_url":"https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/HEAD/sb3_contrib/common/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b720173db270f08b"}},{"code_sha256_prefix":"43887c9edd549830","entry":"discount_cumsum","repo":"vwxyzjn/invalid-action-masking","repo_kind":"official","path":"invalid_action_masking/ppo_10x10.py","file_url":"https://github.com/vwxyzjn/invalid-action-masking/blob/HEAD/invalid_action_masking/ppo_10x10.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43887c9edd549830"}},{"code_sha256_prefix":"530d3c238c8e12f0","entry":"flat_grad","repo":"Stable-Baselines-Team/stable-baselines3-contrib","repo_kind":"listed","path":"sb3_contrib/common/utils.py","file_url":"https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/HEAD/sb3_contrib/common/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"530d3c238c8e12f0"}},{"code_sha256_prefix":"8a3b5a2760bcc10a","entry":"quantile_huber_loss","repo":"Stable-Baselines-Team/stable-baselines3-contrib","repo_kind":"listed","path":"sb3_contrib/common/utils.py","file_url":"https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/HEAD/sb3_contrib/common/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a3b5a2760bcc10a"}},{"code_sha256_prefix":"3ebb420a5d10cc64","entry":"update_mean_var_count_from_moments","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3ebb420a5d10cc64"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}