{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-10624","title":"Reinforcement Learning with Conditional Expectation Reward","arxiv_id":"2603.10624","date":"2026-03-11","proceeding":null,"authors":["Changyi Xiao","Caijun Xu","Yixin Cao"],"abstract":"Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at https://github.com/changyi7231/CER.","url_abs":"https://arxiv.org/abs/2603.10624","url_pdf":"https://arxiv.org/pdf/2603.10624","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2603.10624","atlas_url":"https://app.syntology.ai/?focus=2603.10624","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.10624"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/changyi7231/CER","reach":null}],"summary":{"ran":4,"ran_violates":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":13,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bacda3216fdce102","entry":"exact_match","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"recipe/cer/src/reward_manager.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/recipe/cer/src/reward_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bacda3216fdce102"}},{"code_sha256_prefix":"904946a1f147981f","entry":"extract_boxed_answer","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"recipe/cer/src/data_preparation.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/recipe/cer/src/data_preparation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"904946a1f147981f"}},{"code_sha256_prefix":"5285b1569c3d6597","entry":"extract_solution_answer","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"recipe/cer/src/cer_ray_trainer.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/recipe/cer/src/cer_ray_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5285b1569c3d6597"}},{"code_sha256_prefix":"a6f637988f82514b","entry":"get_prompt","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"recipe/cer/src/data_preparation.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/recipe/cer/src/data_preparation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6f637988f82514b"}},{"code_sha256_prefix":"735e93af30b15fa9","entry":"is_ipv4","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/utils/net_utils.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"735e93af30b15fa9"}},{"code_sha256_prefix":"25f0ea3f460f6ce1","entry":"unpad_dataproto","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/protocol.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25f0ea3f460f6ce1"}},{"code_sha256_prefix":"c9d58036da0286f2","entry":"get_generation_config","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9d58036da0286f2"}},{"code_sha256_prefix":"59904a8955b34252","entry":"get_huggingface_actor_config","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59904a8955b34252"}},{"code_sha256_prefix":"acf240079d434746","entry":"get_weight_loader","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acf240079d434746"}},{"code_sha256_prefix":"55f217ce71432b58","entry":"get_weight_saver","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55f217ce71432b58"}},{"code_sha256_prefix":"705afe175ad8e404","entry":"is_ipv6","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/utils/net_utils.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"705afe175ad8e404"}},{"code_sha256_prefix":"1853101234cf952e","entry":"load_state_dict_to_megatron_gptmodel","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/models/mcore/loader.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/models/mcore/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1853101234cf952e"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"changyi7231/CER","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/changyi7231/CER/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}