{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparse-rewards-can-self-train-dialogue-agents","title":"Sparse Rewards Can Self-Train Dialogue Agents","arxiv_id":"2409.04617","date":"2024-09-06","proceeding":null,"authors":["Barrett Martin Lattimer","Varun Gangal","Ryan Mcdonald","Yi Yang"],"abstract":"Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub at https://github.com/asappresearch/josh-llm-simulation-training","url_abs":"https://arxiv.org/abs/2409.04617v2","url_pdf":"https://arxiv.org/pdf/2409.04617v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparse-rewards-can-self-train-dialogue-agents","repo_url":"https://github.com/asappresearch/josh-llm-simulation-training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.04617","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.04617"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/asappresearch/josh-llm-simulation-training","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"718f019457eaab1e","entry":"clean_good_utterance","repo":"asappresearch/josh-llm-simulation-training","repo_kind":"official","path":"josh_train/finetune/finetune_agent.py","file_url":"https://github.com/asappresearch/josh-llm-simulation-training/blob/HEAD/josh_train/finetune/finetune_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718f019457eaab1e"}},{"code_sha256_prefix":"f45224b2ea9baa91","entry":"parse_agent_message","repo":"asappresearch/josh-llm-simulation-training","repo_kind":"official","path":"josh_train/finetune/finetune_agent.py","file_url":"https://github.com/asappresearch/josh-llm-simulation-training/blob/HEAD/josh_train/finetune/finetune_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f45224b2ea9baa91"}},{"code_sha256_prefix":"c75f9362094ea3ef","entry":"request_openai","repo":"asappresearch/josh-llm-simulation-training","repo_kind":"official","path":"josh_train/utils.py","file_url":"https://github.com/asappresearch/josh-llm-simulation-training/blob/HEAD/josh_train/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c75f9362094ea3ef"}},{"code_sha256_prefix":"a23ca1b7c141ae60","entry":"trim_user_msg","repo":"asappresearch/josh-llm-simulation-training","repo_kind":"official","path":"josh_train/async_josh.py","file_url":"https://github.com/asappresearch/josh-llm-simulation-training/blob/HEAD/josh_train/async_josh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a23ca1b7c141ae60"}},{"code_sha256_prefix":"6d3738cda068547c","entry":"compute_cost","repo":"asappresearch/josh-llm-simulation-training","repo_kind":"official","path":"josh_train/utils.py","file_url":"https://github.com/asappresearch/josh-llm-simulation-training/blob/HEAD/josh_train/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d3738cda068547c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}