{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-23863","title":"SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models","arxiv_id":"2509.23863","date":"2025-09-28","proceeding":"ICLR","authors":["Ziyi Yang","Weizhou Shen","Chenliang Li","Ruijun Chen","Fanqi Wan","Ming Yan","Xiaojun Quan","Fei Huang"],"abstract":"Progress in long-context reasoning for large language models (LLMs) has lagged behind other recent advances. This gap arises not only from the intrinsic difficulty of processing long texts, but also from the scarcity of reliable human annotations and programmatically verifiable reward signals. In this paper, we propose SPELL, a multi-role self-play reinforcement learning framework that enables scalable, label-free optimization for long-context reasoning. SPELL integrates three cyclical roles-questioner, responder, and verifier-within a single model to enable continual self-improvement. The questioner generates questions from raw documents paired with reference answers; the responder learns to solve these questions based on the documents; and the verifier evaluates semantic equivalence between the responder's output and the questioner's reference answer, producing reward signals to guide continual training. To stabilize training, we introduce an automated curriculum that gradually increases document length and a reward function that adapts question difficulty to the model's evolving capabilities. Extensive experiments on six long-context benchmarks show that SPELL consistently improves performance across diverse LLMs and outperforms equally sized models fine-tuned on large-scale annotated data. Notably, SPELL achieves an average 7.6-point gain in pass@8 on the strong reasoning model Qwen3-30B-A3B-Thinking, raising its performance ceiling and showing promise for scaling to even more capable models. Our code is available at https://github.com/Tongyi-Zhiwen/Qwen-Doc.","url_abs":"https://arxiv.org/abs/2509.23863","url_pdf":"https://arxiv.org/pdf/2509.23863","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2509.23863","atlas_url":"https://app.syntology.ai/?focus=2509.23863","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.23863"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Tongyi-Zhiwen/Qwen-Doc","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"4150e89fe1921102","entry":"questioner_mapping_function","repo":"Tongyi-Zhiwen/Qwen-Doc","repo_kind":"found_in_text","path":"SPELL/verl/verl/trainer/ppo/core_algos.py","file_url":"https://github.com/Tongyi-Zhiwen/Qwen-Doc/blob/HEAD/SPELL/verl/verl/trainer/ppo/core_algos.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4150e89fe1921102"}},{"code_sha256_prefix":"6b9eb4f0ec9b8457","entry":"register_adv_est","repo":"Tongyi-Zhiwen/Qwen-Doc","repo_kind":"found_in_text","path":"SPELL/verl/verl/trainer/ppo/core_algos.py","file_url":"https://github.com/Tongyi-Zhiwen/Qwen-Doc/blob/HEAD/SPELL/verl/verl/trainer/ppo/core_algos.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b9eb4f0ec9b8457"}},{"code_sha256_prefix":"b62f6ef15cf876cd","entry":"AdvantageEstimator","repo":"Tongyi-Zhiwen/Qwen-Doc","repo_kind":"found_in_text","path":"SPELL/verl/verl/trainer/ppo/core_algos.py","file_url":"https://github.com/Tongyi-Zhiwen/Qwen-Doc/blob/HEAD/SPELL/verl/verl/trainer/ppo/core_algos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b62f6ef15cf876cd"}},{"code_sha256_prefix":"d0a70c2ef669d178","entry":"compute_spell_grpo_outcome_advantage","repo":"Tongyi-Zhiwen/Qwen-Doc","repo_kind":"found_in_text","path":"SPELL/verl/verl/trainer/ppo/core_algos.py","file_url":"https://github.com/Tongyi-Zhiwen/Qwen-Doc/blob/HEAD/SPELL/verl/verl/trainer/ppo/core_algos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0a70c2ef669d178"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}