{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reconciling-spatial-and-temporal-abstractions","title":"Reconciling Spatial and Temporal Abstractions for Goal Representation","arxiv_id":"2401.09870","date":"2024-01-18","proceeding":null,"authors":["Mehdi Zadem","Sergio Mover","Sao Mai Nguyen"],"abstract":"Goal representation affects the performance of Hierarchical Reinforcement Learning (HRL) algorithms by decomposing the complex learning problem into easier subtasks. Recent studies show that representations that preserve temporally abstract environment dynamics are successful in solving difficult problems and provide theoretical guarantees for optimality. These methods however cannot scale to tasks where environment dynamics increase in complexity i.e. the temporally abstract transition relations depend on larger number of variables. On the other hand, other efforts have tried to use spatial abstraction to mitigate the previous issues. Their limitations include scalability to high dimensional environments and dependency on prior knowledge. In this paper, we propose a novel three-layer HRL algorithm that introduces, at different levels of the hierarchy, both a spatial and a temporal goal abstraction. We provide a theoretical study of the regret bounds of the learned policies. We evaluate the approach on complex continuous control tasks, demonstrating the effectiveness of spatial and temporal abstractions learned by this approach. Find open-source code at https://github.com/cosynus-lix/STAR.","url_abs":"https://arxiv.org/abs/2401.09870v2","url_pdf":"https://arxiv.org/pdf/2401.09870v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reconciling-spatial-and-temporal-abstractions","repo_url":"https://github.com/cosynus-lix/STAR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"hierarchical-reinforcement-learning","task_name":"Hierarchical Reinforcement Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"clipped-double-q-learning","method_name":"Clipped Double Q-learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"td3","method_name":"TD3"},{"method_slug":"target-policy-smoothing","method_name":"Target Policy Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/hierarchical-reinforcement-learning-on-ant","task":"Hierarchical Reinforcement Learning","dataset":"Ant + Maze","model":"STAR","rank_in_archive_order":1,"of":1,"metrics":{"Return":"0.85"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.09870","atlas_url":"https://app.syntology.ai/?focus=2401.09870","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09870"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cosynus-lix/STAR","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran":3,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"4938b585d0d7b67c","entry":"get_tensor","repo":"cosynus-lix/STAR","repo_kind":"official","path":"star/agents.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/star/agents.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4938b585d0d7b67c"}},{"code_sha256_prefix":"061cfe8d8746d43a","entry":"make_epsilon_greedy_policy","repo":"cosynus-lix/STAR","repo_kind":"official","path":"star/agents.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/star/agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"061cfe8d8746d43a"}},{"code_sha256_prefix":"b518b50864d18ead","entry":"q_inv","repo":"cosynus-lix/STAR","repo_kind":"official","path":"envs/ant.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/envs/ant.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b518b50864d18ead"}},{"code_sha256_prefix":"e859483773f4185f","entry":"transform","repo":"cosynus-lix/STAR","repo_kind":"official","path":"region_draw.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/region_draw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e859483773f4185f"}},{"code_sha256_prefix":"6509922bd56726e6","entry":"var","repo":"cosynus-lix/STAR","repo_kind":"official","path":"star/agents.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/star/agents.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6509922bd56726e6"}},{"code_sha256_prefix":"f5f71585129097c1","entry":"calculate_cube_colors","repo":"cosynus-lix/STAR","repo_kind":"official","path":"region_draw.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/region_draw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5f71585129097c1"}},{"code_sha256_prefix":"cde2ae8e18dc91d3","entry":"create_gather_env","repo":"cosynus-lix/STAR","repo_kind":"official","path":"envs/create_gather_env.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/envs/create_gather_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cde2ae8e18dc91d3"}},{"code_sha256_prefix":"ead6fe1a542c22a7","entry":"create_maze_env","repo":"cosynus-lix/STAR","repo_kind":"official","path":"envs/create_maze_env.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/envs/create_maze_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ead6fe1a542c22a7"}},{"code_sha256_prefix":"658b798591505f18","entry":"plot","repo":"cosynus-lix/STAR","repo_kind":"official","path":"plotting.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"658b798591505f18"}},{"code_sha256_prefix":"b5987afb2eb91889","entry":"q_mult","repo":"cosynus-lix/STAR","repo_kind":"official","path":"envs/ant.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/envs/ant.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5987afb2eb91889"}},{"code_sha256_prefix":"edc704754c53dba1","entry":"read_results","repo":"cosynus-lix/STAR","repo_kind":"official","path":"plotting.py","file_url":"https://github.com/cosynus-lix/STAR/blob/HEAD/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"edc704754c53dba1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}