{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spa-rl-reinforcing-llm-agents-via-stepwise","title":"SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution","arxiv_id":"2505.20732","date":"2025-05-27","proceeding":null,"authors":["Hanlin Wang","Chak Tou Leong","Jiashuo Wang","Jian Wang","Wenjie Li"],"abstract":"Reinforcement learning (RL) holds significant promise for training LLM agents to handle complex, goal-oriented tasks that require multi-step interactions with external environments. However, a critical challenge when applying RL to these agentic tasks arises from delayed rewards: feedback signals are typically available only after the entire task is completed. This makes it non-trivial to assign delayed rewards to earlier actions, providing insufficient guidance regarding environmental constraints and hindering agent training. In this work, we draw on the insight that the ultimate completion of a task emerges from the cumulative progress an agent makes across individual steps. We propose Stepwise Progress Attribution (SPA), a general reward redistribution framework that decomposes the final reward into stepwise contributions, each reflecting its incremental progress toward overall task completion. To achieve this, we train a progress estimator that accumulates stepwise contributions over a trajectory to match the task completion. During policy optimization, we combine the estimated per-step contribution with a grounding signal for actions executed in the environment as the fine-grained, intermediate reward for effective agent training. Extensive experiments on common agent benchmarks (including Webshop, ALFWorld, and VirtualHome) demonstrate that SPA consistently outperforms the state-of-the-art method in both success rate (+2.5\\% on average) and grounding accuracy (+1.9\\% on average). Further analyses demonstrate that our method remarkably provides more effective intermediate rewards for RL training. Our code is available at https://github.com/WangHanLinHenry/SPA-RL-Agent.","url_abs":"https://arxiv.org/abs/2505.20732v1","url_pdf":"https://arxiv.org/pdf/2505.20732v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spa-rl-reinforcing-llm-agents-via-stepwise","repo_url":"https://github.com/wanghanlinhenry/spa-rl-agent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.20732","atlas_url":"https://app.syntology.ai/?focus=2505.20732","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.20732"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wanghanlinhenry/spa-rl-agent","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"da76a1cb08f71e53","entry":"process_format","repo":"wanghanlinhenry/spa-rl-agent","repo_kind":"official","path":"eval_agent/main_vh.py","file_url":"https://github.com/wanghanlinhenry/spa-rl-agent/blob/HEAD/eval_agent/main_vh.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da76a1cb08f71e53"}},{"code_sha256_prefix":"041bad03a9d3bad1","entry":"prompt_with_icl","repo":"wanghanlinhenry/spa-rl-agent","repo_kind":"official","path":"eval_agent/prompt/templates.py","file_url":"https://github.com/wanghanlinhenry/spa-rl-agent/blob/HEAD/eval_agent/prompt/templates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"041bad03a9d3bad1"}},{"code_sha256_prefix":"241b9f5e2a386a3d","entry":"remove_duplicate_edge","repo":"wanghanlinhenry/spa-rl-agent","repo_kind":"official","path":"eval_agent/main_vh.py","file_url":"https://github.com/wanghanlinhenry/spa-rl-agent/blob/HEAD/eval_agent/main_vh.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"241b9f5e2a386a3d"}},{"code_sha256_prefix":"91b0f8598b261907","entry":"step","repo":"wanghanlinhenry/spa-rl-agent","repo_kind":"official","path":"eval_agent/utils/replace_sciworld_score.py","file_url":"https://github.com/wanghanlinhenry/spa-rl-agent/blob/HEAD/eval_agent/utils/replace_sciworld_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91b0f8598b261907"}},{"code_sha256_prefix":"dfb17eb3e6eaed35","entry":"process_ob","repo":"wanghanlinhenry/spa-rl-agent","repo_kind":"official","path":"eval_agent/envs/alfworld_env.py","file_url":"https://github.com/wanghanlinhenry/spa-rl-agent/blob/HEAD/eval_agent/envs/alfworld_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb17eb3e6eaed35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}