{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/plug-and-play-policy-planner-for-large","title":"Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents","arxiv_id":"2311.00262","date":"2023-11-01","proceeding":null,"authors":["Yang Deng","Wenxuan Zhang","Wai Lam","See-Kiong Ng","Tat-Seng Chua"],"abstract":"Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in as a plug-and-play dialogue policy planner, named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data as well as reinforcement learning from goal-oriented AI feedback with dynamic interaction data collected by the LLM-based self-play simulation. In this manner, the LLM-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning capability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.","url_abs":"https://arxiv.org/abs/2311.00262v2","url_pdf":"https://arxiv.org/pdf/2311.00262v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"plug-and-play-policy-planner-for-large","repo_url":"https://github.com/dengyang17/ppdpp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.00262","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.00262"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dengyang17/ppdpp","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1},"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"9548d4654fac9295","entry":"CBMessages","repo":"dengyang17/ppdpp","repo_kind":"official","path":"prompt.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9548d4654fac9295"}},{"code_sha256_prefix":"aba8c48b0ee40f8b","entry":"CIMAMessages","repo":"dengyang17/ppdpp","repo_kind":"official","path":"prompt.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aba8c48b0ee40f8b"}},{"code_sha256_prefix":"3e7f40b21d97cc7d","entry":"ESConvMessages","repo":"dengyang17/ppdpp","repo_kind":"official","path":"prompt.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e7f40b21d97cc7d"}},{"code_sha256_prefix":"f6b7e72b3271c4d4","entry":"collate_fn","repo":"dengyang17/ppdpp","repo_kind":"official","path":"sft.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/sft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6b7e72b3271c4d4"}},{"code_sha256_prefix":"77f69af078e09830","entry":"mean_pooling","repo":"dengyang17/llm-proactive","repo_kind":"community","path":"topkg/chatgpt_self_play.py","file_url":"https://github.com/dengyang17/llm-proactive/blob/HEAD/topkg/chatgpt_self_play.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77f69af078e09830"}},{"code_sha256_prefix":"3f4a343ff94d31a0","entry":"query_openai_model","repo":"dengyang17/ppdpp","repo_kind":"official","path":"env.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/env.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f4a343ff94d31a0"}},{"code_sha256_prefix":"28e7c08042bacaaf","entry":"read_pkl","repo":"dengyang17/ppdpp","repo_kind":"official","path":"data_reader.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/data_reader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28e7c08042bacaaf"}},{"code_sha256_prefix":"8e3538aa7c7d9315","entry":"set_cuda","repo":"dengyang17/ppdpp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e3538aa7c7d9315"}},{"code_sha256_prefix":"74a8c1a0129c43ba","entry":"load_dataset","repo":"dengyang17/ppdpp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dengyang17/ppdpp/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74a8c1a0129c43ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}