{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dra-grpo-exploring-diversity-aware-reward","title":"DRA-GRPO: Exploring Diversity-Aware Reward Adjustment for R1-Zero-Like Training of Large Language Models","arxiv_id":"2505.09655","date":"2025-05-14","proceeding":null,"authors":["Xiwen Chen","Wenhui Zhu","Peijie Qiu","Xuanzhao Dong","Hao Wang","Haiyu Wu","Huayu Li","Aristeidis Sotiras","Yalin Wang","Abolfazl Razi"],"abstract":"Recent advances in reinforcement learning for language model post-training, such as Group Relative Policy Optimization (GRPO), have shown promise in low-resource settings. However, GRPO typically relies on solution-level and scalar reward signals that fail to capture the semantic diversity among sampled completions. This leads to what we identify as a diversity-quality inconsistency, where distinct reasoning paths may receive indistinguishable rewards. To address this limitation, we propose $\\textit{Diversity-aware Reward Adjustment}$ (DRA), a method that explicitly incorporates semantic diversity into the reward computation. DRA uses Submodular Mutual Information (SMI) to downweight redundant completions and amplify rewards for diverse ones. This encourages better exploration during learning, while maintaining stable exploitation of high-quality samples. Our method integrates seamlessly with both GRPO and its variant DR.~GRPO, resulting in $\\textit{DRA-GRPO}$ and $\\textit{DGA-DR.~GRPO}$. We evaluate our method on five mathematical reasoning benchmarks and find that it outperforms recent strong baselines. It achieves state-of-the-art performance with an average accuracy of 58.2%, using only 7,000 fine-tuning samples and a total training cost of approximately $55. The code is available at https://github.com/xiwenc1/DRA-GRPO.","url_abs":"https://arxiv.org/abs/2505.09655v2","url_pdf":"https://arxiv.org/pdf/2505.09655v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dra-grpo-exploring-diversity-aware-reward","repo_url":"https://github.com/xiwenc1/dra-grpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"dra","method_name":"DRA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.09655","atlas_url":"https://app.syntology.ai/?focus=2505.09655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.09655"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiwenc1/dra-grpo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dfd0ecd9cb25af69","entry":"flatten_dict","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/core.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfd0ecd9cb25af69"}},{"code_sha256_prefix":"4f7075aac2163cbc","entry":"masked_mean","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/core.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f7075aac2163cbc"}},{"code_sha256_prefix":"67607956afbb2a83","entry":"masked_var","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/core.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67607956afbb2a83"}},{"code_sha256_prefix":"0a9f60618bd16d86","entry":"tag_count_reward","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/rewards.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/rewards.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a9f60618bd16d86"}},{"code_sha256_prefix":"53017c7a8e570221","entry":"apply_chat_template","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/data_utils.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53017c7a8e570221"}},{"code_sha256_prefix":"0a5c0b6fb53cd48c","entry":"format_reward","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/rewards.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/rewards.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a5c0b6fb53cd48c"}},{"code_sha256_prefix":"3c42df7a6e98e89c","entry":"is_conversational","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/data_utils.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c42df7a6e98e89c"}},{"code_sha256_prefix":"c695ba9ac4cc3b93","entry":"maybe_apply_chat_template","repo":"xiwenc1/dra-grpo","repo_kind":"official","path":"src/open_r1/trl/data_utils.py","file_url":"https://github.com/xiwenc1/dra-grpo/blob/HEAD/src/open_r1/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c695ba9ac4cc3b93"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}