{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/effective-diversity-in-population-based","title":"Effective Diversity in Population Based Reinforcement Learning","arxiv_id":"2002.00632","date":"2020-02-03","proceeding":"NeurIPS 2020 12","authors":["Jack Parker-Holder","Aldo Pacchiano","Krzysztof Choromanski","Stephen Roberts"],"abstract":"Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.","url_abs":"https://arxiv.org/abs/2002.00632v3","url_pdf":"https://arxiv.org/pdf/2002.00632v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"effective-diversity-in-population-based","repo_url":"https://github.com/jparkerholder/DvD_ES","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"effective-diversity-in-population-based","repo_url":"https://github.com/holounic/DvD-TD3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"point-processes","task_name":"Point Processes"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2002.00632","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.00632"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/holounic/DvD-TD3","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jparkerholder/DvD_ES","reach":null}],"summary":{"ran":6,"ran_fixture":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"01cff9d67cdcaecf","entry":"Actor","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01cff9d67cdcaecf"}},{"code_sha256_prefix":"347f0fe517e50968","entry":"BaseActorCritic","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"347f0fe517e50968"}},{"code_sha256_prefix":"aeecae3f15ea94c6","entry":"Buffer","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aeecae3f15ea94c6"}},{"code_sha256_prefix":"b97e9e0f24a99b33","entry":"Critic","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b97e9e0f24a99b33"}},{"code_sha256_prefix":"71b8dc99038de04a","entry":"DiversityLoss","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71b8dc99038de04a"}},{"code_sha256_prefix":"ea4ac6af2af793cd","entry":"TD3","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea4ac6af2af793cd"}},{"code_sha256_prefix":"9f4ef0e29eb8bd25","entry":"get_det","repo":"jparkerholder/DvD_ES","repo_kind":"official","path":"es.py","file_url":"https://github.com/jparkerholder/DvD_ES/blob/HEAD/es.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f4ef0e29eb8bd25"}},{"code_sha256_prefix":"808b1cbcf8ffbd49","entry":"get_experiment","repo":"jparkerholder/DvD_ES","repo_kind":"official","path":"experiments.py","file_url":"https://github.com/jparkerholder/DvD_ES/blob/HEAD/experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"808b1cbcf8ffbd49"}},{"code_sha256_prefix":"954aa28753459d3b","entry":"normalize2","repo":"jparkerholder/DvD_ES","repo_kind":"official","path":"es.py","file_url":"https://github.com/jparkerholder/DvD_ES/blob/HEAD/es.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"954aa28753459d3b"}},{"code_sha256_prefix":"a72924a3e9838a33","entry":"unpack_batch","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a72924a3e9838a33"}},{"code_sha256_prefix":"34d3b149455f4fee","entry":"DvDTD3","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34d3b149455f4fee"}},{"code_sha256_prefix":"ea8c4add89ea88fb","entry":"soft_update","repo":"holounic/DvD-TD3","repo_kind":"listed","path":"td3/algorithm.py","file_url":"https://github.com/holounic/DvD-TD3/blob/HEAD/td3/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea8c4add89ea88fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}