{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/query-policy-misalignment-in-preference-based","title":"Query-Policy Misalignment in Preference-Based Reinforcement Learning","arxiv_id":"2305.17400","date":"2023-05-27","proceeding":null,"authors":["Xiao Hu","Jianxiong Li","Xianyuan Zhan","Qing-Shan Jia","Ya-Qin Zhang"],"abstract":"Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.","url_abs":"https://arxiv.org/abs/2305.17400v3","url_pdf":"https://arxiv.org/pdf/2305.17400v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"query-policy-misalignment-in-preference-based","repo_url":"https://github.com/huxiao09/qpa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.17400","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.17400"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huxiao09/QPA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huxiao09/qpa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fe29d8cf8d1bbbeb","entry":"RewardModel","repo":"huxiao09/qpa","repo_kind":"official","path":"reward_model.py","file_url":"https://github.com/huxiao09/qpa/blob/HEAD/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe29d8cf8d1bbbeb"}},{"code_sha256_prefix":"c2ac288d85bea137","entry":"compute_smallest_dist","repo":"huxiao09/QPA","repo_kind":"official","path":"reward_model.py","file_url":"https://github.com/huxiao09/QPA/blob/HEAD/reward_model.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2ac288d85bea137"}},{"code_sha256_prefix":"a17d99b8eda680ff","entry":"gen_net","repo":"huxiao09/QPA","repo_kind":"official","path":"reward_model.py","file_url":"https://github.com/huxiao09/QPA/blob/HEAD/reward_model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a17d99b8eda680ff"}},{"code_sha256_prefix":"0f98617f44c6afff","entry":"KCenterGreedy","repo":"huxiao09/QPA","repo_kind":"official","path":"reward_model.py","file_url":"https://github.com/huxiao09/QPA/blob/HEAD/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f98617f44c6afff"}},{"code_sha256_prefix":"115ab9ed9d356045","entry":"compute_state_entropy","repo":"huxiao09/QPA","repo_kind":"official","path":"agent/sac.py","file_url":"https://github.com/huxiao09/QPA/blob/HEAD/agent/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"115ab9ed9d356045"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}