{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/toward-optimal-llm-alignments-using-two","title":"Toward Optimal LLM Alignments Using Two-Player Games","arxiv_id":"2406.10977","date":"2024-06-16","proceeding":null,"authors":["Rui Zheng","Hongyi Guo","Zhihan Liu","Xiaoying Zhang","Yuanshun Yao","Xiaojun Xu","Zhaoran Wang","Zhiheng Xi","Tao Gui","Qi Zhang","Xuanjing Huang","Hang Li","Yang Liu"],"abstract":"The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. The adversarial agent's task at each step is to generate prompts that expose the weakness of the defensive agent. In return, the defensive agent seeks to improve its responses to these newly identified prompts it struggled with, based on feedback from the reward model. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.","url_abs":"https://arxiv.org/abs/2406.10977v1","url_pdf":"https://arxiv.org/pdf/2406.10977v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"toward-optimal-llm-alignments-using-two","repo_url":"https://github.com/ruizheng20/gpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.10977","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10977"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruizheng20/gpo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4ac3d9d133a3734b","entry":"build_prompt_history","repo":"ruizheng20/gpo","repo_kind":"official","path":"gpo/gpo_datahelper.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/gpo/gpo_datahelper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ac3d9d133a3734b"}},{"code_sha256_prefix":"f6d83385de70f664","entry":"get_adv_separate_prompt","repo":"ruizheng20/gpo","repo_kind":"official","path":"gpo/gpo_datahelper.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/gpo/gpo_datahelper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6d83385de70f664"}},{"code_sha256_prefix":"e1507eef3b40426e","entry":"get_global_statistics","repo":"ruizheng20/gpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1507eef3b40426e"}},{"code_sha256_prefix":"31961e93e883be23","entry":"get_separate_prompt","repo":"ruizheng20/gpo","repo_kind":"official","path":"gpo/gpo_datahelper.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/gpo/gpo_datahelper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31961e93e883be23"}},{"code_sha256_prefix":"8a9c8c138ba6f286","entry":"get_separate_prompt","repo":"ruizheng20/gpo","repo_kind":"official","path":"rm/reward_datahelper.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/rm/reward_datahelper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a9c8c138ba6f286"}},{"code_sha256_prefix":"77f69af078e09830","entry":"mean_pooling","repo":"ruizheng20/gpo","repo_kind":"official","path":"gpo/sentence_embed.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/gpo/sentence_embed.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77f69af078e09830"}},{"code_sha256_prefix":"b60c18c9b0c0e819","entry":"whiten","repo":"ruizheng20/gpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b60c18c9b0c0e819"}},{"code_sha256_prefix":"259b0b2755f70e0e","entry":"get_eval_ds_config","repo":"ruizheng20/gpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ruizheng20/gpo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"259b0b2755f70e0e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}