{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rulereasoner-reinforced-rule-based-reasoning","title":"RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling","arxiv_id":"2506.08672","date":"2025-06-10","proceeding":null,"authors":["Yang Liu","Jiaqi Li","Zilong Zheng"],"abstract":"Rule-based reasoning has been acknowledged as one of the fundamental problems in reasoning, while deviations in rule formats, types, and complexity in real-world applications pose severe challenges. Recent studies have shown that large reasoning models (LRMs) have remarkable reasoning capabilities, and their performance is substantially enhanced by reinforcement learning (RL). However, it remains an open question whether small reasoning models (SRMs) can learn rule-based reasoning effectively with robust generalization across diverse tasks and domains. To address this, we introduce Reinforced Rule-based Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct rule-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples each training batch by updating the sampling weights of different domains based on historical rewards. This facilitates domain augmentation and flexible online learning schedules for RL, obviating the need for pre-hoc human-engineered mix-training recipes used in existing methods. Empirical evaluations on in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that RuleReasoner outperforms frontier LRMs by a significant margin ($\\Delta$4.1% average points on eight ID tasks and $\\Delta$10.4% average points on three OOD tasks over OpenAI-o1). Notably, our approach also exhibits higher computational efficiency compared to prior dynamic sampling methods for RL.","url_abs":"https://arxiv.org/abs/2506.08672v1","url_pdf":"https://arxiv.org/pdf/2506.08672v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rulereasoner-reinforced-rule-based-reasoning","repo_url":"https://github.com/bigai-nlco/rulereasoner","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.08672","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.08672"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigai-nlco/rulereasoner","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_fixture":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":13,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"735e93af30b15fa9","entry":"is_ipv4","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/utils/net_utils.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735e93af30b15fa9"}},{"code_sha256_prefix":"c30b2a96103abd2c","entry":"reward_func","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/recipe/r1/reward_score.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c30b2a96103abd2c"}},{"code_sha256_prefix":"21331a58f93375e4","entry":"union_tensor_dict","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/protocol.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21331a58f93375e4"}},{"code_sha256_prefix":"25f0ea3f460f6ce1","entry":"unpad_dataproto","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/protocol.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25f0ea3f460f6ce1"}},{"code_sha256_prefix":"b6512040439cb7a2","entry":"get_custom_reward_fn","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/recipe/dapo/src/main_dapo.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/recipe/dapo/src/main_dapo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6512040439cb7a2"}},{"code_sha256_prefix":"68aa34a57b353cae","entry":"get_generation_config","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/utils/model.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68aa34a57b353cae"}},{"code_sha256_prefix":"9411d1fb7e4f68e5","entry":"get_huggingface_actor_config","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/utils/model.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9411d1fb7e4f68e5"}},{"code_sha256_prefix":"acf240079d434746","entry":"get_weight_loader","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/models/weight_loader_registry.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acf240079d434746"}},{"code_sha256_prefix":"1008c22d29f85015","entry":"get_weight_saver","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/models/weight_loader_registry.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1008c22d29f85015"}},{"code_sha256_prefix":"705afe175ad8e404","entry":"is_ipv6","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/utils/net_utils.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"705afe175ad8e404"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"bigai-nlco/rulereasoner","repo_kind":"official","path":"verl/verl/utils/model.py","file_url":"https://github.com/bigai-nlco/rulereasoner/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}