{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spar-self-play-with-tree-search-refinement-to","title":"SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models","arxiv_id":"2412.11605","date":"2024-12-16","proceeding":null,"authors":["Jiale Cheng","Xiao Liu","Cunxiang Wang","Xiaotao Gu","Yida Lu","Dan Zhang","Yuxiao Dong","Jie Tang","Hongning Wang","Minlie Huang"],"abstract":"Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output. Such an ability is well-suited for and often optimized by preference learning. However, existing methods often directly sample multiple independent responses from the model when creating preference pairs. Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following. In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations. Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. Furthermore, SPaR demonstrates promising scalability and transferability, greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how inference scaling in tree search would impact model performance. Our code and data are publicly available at https://github.com/thu-coai/SPaR.","url_abs":"https://arxiv.org/abs/2412.11605v1","url_pdf":"https://arxiv.org/pdf/2412.11605v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spar-self-play-with-tree-search-refinement-to","repo_url":"https://github.com/thu-coai/spar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.11605","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.11605"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/thu-coai/SPaR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-coai/spar","reach":null}],"summary":{"ran_draft_wrong":1,"ran":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1318354de90ed3ee","entry":"BFS","repo":"thu-coai/spar","repo_kind":"official","path":"src/ToT/bfs.py","file_url":"https://github.com/thu-coai/spar/blob/HEAD/src/ToT/bfs.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1318354de90ed3ee"}},{"code_sha256_prefix":"6fd053d873aad5f2","entry":"Node","repo":"thu-coai/spar","repo_kind":"official","path":"src/ToT/bfs.py","file_url":"https://github.com/thu-coai/spar/blob/HEAD/src/ToT/bfs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fd053d873aad5f2"}},{"code_sha256_prefix":"b86bf8965e826c29","entry":"rand_select","repo":"thu-coai/spar","repo_kind":"official","path":"src/ToT/bfs.py","file_url":"https://github.com/thu-coai/spar/blob/HEAD/src/ToT/bfs.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b86bf8965e826c29"}},{"code_sha256_prefix":"6af0652b632cb33b","entry":"chat_gpt","repo":"thu-coai/SPaR","repo_kind":"official","path":"src/tree_search.py","file_url":"https://github.com/thu-coai/SPaR/blob/HEAD/src/tree_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6af0652b632cb33b"}},{"code_sha256_prefix":"b91792318ca93632","entry":"multi_process_chat_gpt","repo":"thu-coai/SPaR","repo_kind":"official","path":"src/tree_search.py","file_url":"https://github.com/thu-coai/SPaR/blob/HEAD/src/tree_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b91792318ca93632"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}