{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ragen-understanding-self-evolution-in-llm","title":"RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning","arxiv_id":"2504.20073","date":"2025-04-24","proceeding":null,"authors":["Zihan Wang","Kangrui Wang","Qineng Wang","Pingyue Zhang","Linjie Li","Zhengyuan Yang","Kefan Yu","Minh Nhat Nguyen","Licheng Liu","Eli Gottlieb","Monica Lam","Yiping Lu","Kyunghyun Cho","Jiajun Wu","Li Fei-Fei","Lijuan Wang","Yejin Choi","Manling Li"],"abstract":"Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on three stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and decoupled clipping. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN.","url_abs":"https://arxiv.org/abs/2504.20073v1","url_pdf":"https://arxiv.org/pdf/2504.20073v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ragen-understanding-self-evolution-in-llm","repo_url":"https://github.com/ragen-ai/ragen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"ragen-understanding-self-evolution-in-llm","repo_url":"https://github.com/zihanwang314/ragen","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2504.20073","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.20073"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zihanwang314/ragen","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ragen-ai/ragen","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dd9a34fc6888e4a5","entry":"compute_grpo_outcome_advantage","repo":"ragen-ai/ragen","repo_kind":"official","path":"ragen/trainer/core_algos.py","file_url":"https://github.com/ragen-ai/ragen/blob/HEAD/ragen/trainer/core_algos.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd9a34fc6888e4a5"}},{"code_sha256_prefix":"0f09f1a969597bf1","entry":"get_masks_and_scores","repo":"ragen-ai/ragen","repo_kind":"official","path":"ragen/llm_agent/ctx_manager.py","file_url":"https://github.com/ragen-ai/ragen/blob/HEAD/ragen/llm_agent/ctx_manager.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f09f1a969597bf1"}},{"code_sha256_prefix":"f849fb5714bbd7a6","entry":"get_special_tokens","repo":"ragen-ai/ragen","repo_kind":"official","path":"ragen/llm_agent/ctx_manager.py","file_url":"https://github.com/ragen-ai/ragen/blob/HEAD/ragen/llm_agent/ctx_manager.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f849fb5714bbd7a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}