{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quality-diversity-through-human-feedback","title":"Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization","arxiv_id":"2310.12103","date":"2023-10-18","proceeding":null,"authors":["Li Ding","Jenny Zhang","Jeff Clune","Lee Spector","Joel Lehman"],"abstract":"Reinforcement Learning from Human Feedback (RLHF) has shown potential in qualitative tasks where easily defined performance measures are lacking. However, there are drawbacks when RLHF is commonly used to optimize for average human preferences, especially in generative tasks that demand diverse model responses. Meanwhile, Quality Diversity (QD) algorithms excel at identifying diverse and high-quality solutions but often rely on manually crafted diversity metrics. This paper introduces Quality Diversity through Human Feedback (QDHF), a novel approach that progressively infers diversity metrics from human judgments of similarity among solutions, thereby enhancing the applicability and effectiveness of QD algorithms in complex and open-ended domains. Empirical studies show that QDHF significantly outperforms state-of-the-art methods in automatic diversity discovery and matches the efficacy of QD with manually crafted diversity metrics on standard benchmarks in robotics and reinforcement learning. Notably, in open-ended generative tasks, QDHF substantially enhances the diversity of text-to-image generation from a diffusion model and is more favorably received in user studies. We conclude by analyzing QDHF's scalability, robustness, and quality of derived diversity metrics, emphasizing its strength in open-ended optimization tasks. Code and tutorials are available at https://liding.info/qdhf.","url_abs":"https://arxiv.org/abs/2310.12103v3","url_pdf":"https://arxiv.org/pdf/2310.12103v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quality-diversity-through-human-feedback","repo_url":"https://github.com/ld-ing/qdhf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"robotic-grasping","task_name":"Robotic Grasping"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.12103","atlas_url":"https://app.syntology.ai/?focus=2310.12103","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ld-ing/qdhf","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cc9d79fa0ef76213","entry":"fit_ae","repo":"ld-ing/qdhf","repo_kind":"official","path":"arm/utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/arm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc9d79fa0ef76213"}},{"code_sha256_prefix":"04b894983e853cbb","entry":"fit_ae","repo":"ld-ing/qdhf","repo_kind":"official","path":"lsi/lsi_utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/lsi/lsi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04b894983e853cbb"}},{"code_sha256_prefix":"9379db3f2290a2ae","entry":"fit_ae","repo":"ld-ing/qdhf","repo_kind":"official","path":"maze/utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/maze/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9379db3f2290a2ae"}},{"code_sha256_prefix":"bc5b03184ad01f17","entry":"fit_dis_embed","repo":"ld-ing/qdhf","repo_kind":"official","path":"arm/utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/arm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc5b03184ad01f17"}},{"code_sha256_prefix":"9847b9196dde5a2e","entry":"fit_dis_embed","repo":"ld-ing/qdhf","repo_kind":"official","path":"lsi/lsi_utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/lsi/lsi_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9847b9196dde5a2e"}},{"code_sha256_prefix":"e36b31f563696ee5","entry":"fit_dis_embed","repo":"ld-ing/qdhf","repo_kind":"official","path":"maze/utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/maze/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e36b31f563696ee5"}},{"code_sha256_prefix":"bc38843352b4eff1","entry":"fit_pca","repo":"ld-ing/qdhf","repo_kind":"official","path":"arm/utils.py","file_url":"https://github.com/ld-ing/qdhf/blob/HEAD/arm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc38843352b4eff1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}