{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generating-adjacency-constrained-subgoals-in","title":"Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning","arxiv_id":"2006.11485","date":"2020-06-20","proceeding":"NeurIPS 2020 12","authors":["Tianren Zhang","Shangqi Guo","Tian Tan","Xiaolin Hu","Feng Chen"],"abstract":"Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency constraint. We theoretically prove that the proposed adjacency constraint preserves the optimal hierarchical policy in deterministic MDPs, and show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks show that incorporating the adjacency constraint improves the performance of state-of-the-art HRL approaches in both deterministic and stochastic environments.","url_abs":"https://arxiv.org/abs/2006.11485v4","url_pdf":"https://arxiv.org/pdf/2006.11485v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generating-adjacency-constrained-subgoals-in","repo_url":"https://github.com/trzhang0116/HRAC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"hierarchical-reinforcement-learning","task_name":"Hierarchical Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.11485","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.11485"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trzhang0116/HRAC","reach":null}],"summary":{"ran":4,"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"1b8c1dbf0a21cb68","entry":"Actor","repo":"trzhang0116/HRAC","repo_kind":"official","path":"hrac/hrac.py","file_url":"https://github.com/trzhang0116/HRAC/blob/HEAD/hrac/hrac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b8c1dbf0a21cb68"}},{"code_sha256_prefix":"21f95f4c74fe408f","entry":"Critic","repo":"trzhang0116/HRAC","repo_kind":"official","path":"hrac/hrac.py","file_url":"https://github.com/trzhang0116/HRAC/blob/HEAD/hrac/hrac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21f95f4c74fe408f"}},{"code_sha256_prefix":"1fa82929d9feb960","entry":"ManagerActor","repo":"trzhang0116/HRAC","repo_kind":"official","path":"hrac/hrac.py","file_url":"https://github.com/trzhang0116/HRAC/blob/HEAD/hrac/hrac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fa82929d9feb960"}},{"code_sha256_prefix":"240446d9a9b4832d","entry":"ManagerCritic","repo":"trzhang0116/HRAC","repo_kind":"official","path":"hrac/hrac.py","file_url":"https://github.com/trzhang0116/HRAC/blob/HEAD/hrac/hrac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"240446d9a9b4832d"}},{"code_sha256_prefix":"4938b585d0d7b67c","entry":"get_tensor","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4938b585d0d7b67c"}},{"code_sha256_prefix":"6509922bd56726e6","entry":"var","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"6509922bd56726e6"}},{"code_sha256_prefix":"510839c21a607e19","entry":"Manager","repo":"trzhang0116/HRAC","repo_kind":"official","path":"hrac/hrac.py","file_url":"https://github.com/trzhang0116/HRAC/blob/HEAD/hrac/hrac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"510839c21a607e19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}