{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-21473","title":"MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization","arxiv_id":"2510.21473","date":"2025-10-24","proceeding":"NeurIPS","authors":["Chenglong Wang","Yang Gan","Hang Zhou","Chi Hu","Yongyu Mu","Kai Song","Murun Yang","Bei Li","Chunliang Zhang","Tongran Liu","Jingbo Zhu","Zhengtao Yu","Tong Xiao"],"abstract":"Recent advances in diffusion language models (DLMs) have presented a promising alternative to traditional autoregressive large language models (LLMs). However, DLMs still lag behind LLMs in reasoning performance, especially as the number of denoising steps decreases. Our analysis reveals that this shortcoming arises primarily from the independent generation of masked tokens across denoising steps, which fails to capture the token correlation. In this paper, we define two types of token correlation: intra-sequence correlation and inter-sequence correlation, and demonstrate that enhancing these correlations improves reasoning performance. To this end, we propose a Multi-Reward Optimization (MRO) approach, which encourages DLMs to consider the token correlation during the denoising process. More specifically, our MRO approach leverages test-time scaling, reject sampling, and reinforcement learning to directly optimize the token correlation with multiple elaborate rewards. Additionally, we introduce group step and importance sampling strategies to mitigate reward variance and enhance sampling efficiency. Through extensive experiments, we demonstrate that MRO not only improves reasoning performance but also achieves significant sampling speedups while maintaining high performance on reasoning benchmarks.","url_abs":"https://arxiv.org/abs/2510.21473","url_pdf":"https://arxiv.org/pdf/2510.21473","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2510.21473","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.21473"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wangclnlp/MRO","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dbea5358be6b5ad0","entry":"create_score_evaluation_response","repo":"wangclnlp/MRO","repo_kind":"found_in_text","path":"src/llamafactory/api/chat.py","file_url":"https://github.com/wangclnlp/MRO/blob/HEAD/src/llamafactory/api/chat.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbea5358be6b5ad0"}},{"code_sha256_prefix":"52a9ebcf2e9dee87","entry":"dictify","repo":"wangclnlp/MRO","repo_kind":"found_in_text","path":"src/llamafactory/api/common.py","file_url":"https://github.com/wangclnlp/MRO/blob/HEAD/src/llamafactory/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52a9ebcf2e9dee87"}},{"code_sha256_prefix":"57583cf880d95444","entry":"jsonify","repo":"wangclnlp/MRO","repo_kind":"found_in_text","path":"src/llamafactory/api/common.py","file_url":"https://github.com/wangclnlp/MRO/blob/HEAD/src/llamafactory/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57583cf880d95444"}},{"code_sha256_prefix":"1669f517a7370894","entry":"convert_to_float","repo":"wangclnlp/MRO","repo_kind":"found_in_text","path":"diffusion_llm_evaluation/eval_datasets.py","file_url":"https://github.com/wangclnlp/MRO/blob/HEAD/diffusion_llm_evaluation/eval_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1669f517a7370894"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}