{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyar-addressing-discrete-continuous-action","title":"HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation","arxiv_id":"2109.05490","date":"2021-09-12","proceeding":"ICLR 2022 4","authors":["Boyan Li","Hongyao Tang","Yan Zheng","Jianye Hao","Pengyi Li","Zhen Wang","Zhaopeng Meng","Li Wang"],"abstract":"Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.","url_abs":"https://arxiv.org/abs/2109.05490v3","url_pdf":"https://arxiv.org/pdf/2109.05490v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hyar-addressing-discrete-continuous-action","repo_url":"https://github.com/opendilab/DI-engine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.05490","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.05490"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sfujim/TD3","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/cycraig/MP-DQN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/DI-engine","reach":null}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cc8c1a2169afb326","entry":"Agent","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc8c1a2169afb326"}},{"code_sha256_prefix":"5c0ed95b8ea1a2ee","entry":"Memory","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0ed95b8ea1a2ee"}},{"code_sha256_prefix":"c78b71fa01da6fe4","entry":"OrnsteinUhlenbeckActionNoise","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c78b71fa01da6fe4"}},{"code_sha256_prefix":"51ae94deb30e2669","entry":"ParamActor","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51ae94deb30e2669"}},{"code_sha256_prefix":"d7c857e61b5c6f8d","entry":"QActor","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7c857e61b5c6f8d"}},{"code_sha256_prefix":"5731e9a2d6b0a167","entry":"RingBuffer","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5731e9a2d6b0a167"}},{"code_sha256_prefix":"2cfe91624780b68c","entry":"PDQNAgent","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cfe91624780b68c"}},{"code_sha256_prefix":"5cbb6ae99ea39c16","entry":"hard_update_target_network","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cbb6ae99ea39c16"}},{"code_sha256_prefix":"15dd8f092a526ade","entry":"soft_update_target_network","repo":"cycraig/MP-DQN","repo_kind":"found_in_text","path":"agents/pdqn.py","file_url":"https://github.com/cycraig/MP-DQN/blob/HEAD/agents/pdqn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15dd8f092a526ade"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}