{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/constrained-ensemble-exploration-for","title":"Constrained Ensemble Exploration for Unsupervised Skill Discovery","arxiv_id":"2405.16030","date":"2024-05-25","proceeding":null,"authors":["Chenjia Bai","Rushuai Yang","Qiaosheng Zhang","Kang Xu","Yi Chen","Ting Xiao","Xuelong Li"],"abstract":"Unsupervised Reinforcement Learning (RL) provides a promising paradigm for learning useful behaviors via reward-free per-training. Existing methods for unsupervised RL mainly conduct empowerment-driven skill discovery or entropy-based exploration. However, empowerment often leads to static skills, and pure exploration only maximizes the state coverage rather than learning useful behaviors. In this paper, we propose a novel unsupervised RL framework via an ensemble of skills, where each skill performs partition exploration based on the state prototypes. Thus, each skill can explore the clustered area locally, and the ensemble skills maximize the overall state coverage. We adopt state-distribution constraints for the skill occupancy and the desired cluster for learning distinguishable skills. Theoretical analysis is provided for the state entropy and the resulting skill distributions. Based on extensive experiments on several challenging tasks, we find our method learns well-explored ensemble skills and achieves superior performance in various downstream tasks compared to previous methods.","url_abs":"https://arxiv.org/abs/2405.16030v1","url_pdf":"https://arxiv.org/pdf/2405.16030v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"unsupervised-reinforcement-learning","task_name":"Unsupervised Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.16030","atlas_url":"https://app.syntology.ai/?focus=2405.16030","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16030"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Baichenjia/CeSD","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac6616507abf7f64","entry":"compute_apt_reward","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"agent/cesd.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/agent/cesd.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac6616507abf7f64"}},{"code_sha256_prefix":"9f840a22d31a89bc","entry":"episode_len","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"replay_buffer.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f840a22d31a89bc"}},{"code_sha256_prefix":"2478bb95b38ef24f","entry":"grad_norm","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2478bb95b38ef24f"}},{"code_sha256_prefix":"184876e5f4ab6562","entry":"load_episode","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"replay_buffer.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"184876e5f4ab6562"}},{"code_sha256_prefix":"717d02cf517c8633","entry":"make_replay_loader","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"replay_buffer.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"717d02cf517c8633"}},{"code_sha256_prefix":"8708aaca52006e47","entry":"param_norm","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8708aaca52006e47"}},{"code_sha256_prefix":"6244e9922f4e0610","entry":"to_torch","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6244e9922f4e0610"}},{"code_sha256_prefix":"2a76dfe317adeb03","entry":"make_agent","repo":"Baichenjia/CeSD","repo_kind":"found_in_text","path":"finetune.py","file_url":"https://github.com/Baichenjia/CeSD/blob/HEAD/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a76dfe317adeb03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}