{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/monte-carlo-tree-search-boosts-reasoning-via","title":"Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning","arxiv_id":"2405.00451","date":"2024-05-01","proceeding":null,"authors":["Yuxi Xie","Anirudh Goyal","Wenyue Zheng","Min-Yen Kan","Timothy P. Lillicrap","Kenji Kawaguchi","Michael Shieh"],"abstract":"We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to $81.8\\%$ (+$5.9\\%$), $34.7\\%$ (+$5.8\\%$), and $76.4\\%$ (+$15.8\\%$), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO.","url_abs":"https://arxiv.org/abs/2405.00451v2","url_pdf":"https://arxiv.org/pdf/2405.00451v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"monte-carlo-tree-search-boosts-reasoning-via","repo_url":"https://github.com/YuxiXie/MCTS-DPO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"monte-carlo-tree-search-boosts-reasoning-via","repo_url":"https://github.com/codelion/optillm/blob/main/optillm/mcts.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"arc","task_name":"ARC"},{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"alphazero","method_name":"AlphaZero"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.00451","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.00451"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codelion/optillm/blob/main/optillm/mcts.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YuxiXie/MCTS-DPO","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c9a2daaf02cf5c5c","entry":"find_all_linear_names","repo":"YuxiXie/MCTS-DPO","repo_kind":"official","path":"mcts_rl/models/pretrained.py","file_url":"https://github.com/YuxiXie/MCTS-DPO/blob/HEAD/mcts_rl/models/pretrained.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9a2daaf02cf5c5c"}},{"code_sha256_prefix":"9b9e20fb02209913","entry":"str2bool","repo":"YuxiXie/MCTS-DPO","repo_kind":"official","path":"mcts_rl/utils.py","file_url":"https://github.com/YuxiXie/MCTS-DPO/blob/HEAD/mcts_rl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b9e20fb02209913"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}