{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-10101","title":"CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR","arxiv_id":"2603.10101","date":"2026-03-10","proceeding":null,"authors":["Sijia Cui","Pengyu Cheng","Jiajun Song","Yongbo Gai","Guojun Zhang","Zhechao Yu","Jianhe Lin","Xiaoxi Jiang","Guanjun Jiang"],"abstract":"Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.","url_abs":"https://arxiv.org/abs/2603.10101","url_pdf":"https://arxiv.org/pdf/2603.10101","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.10101","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.10101"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Qwen-Applications/CLIPO","reach":null}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"8896f4e2bafe8abc","entry":"generate_config_from_args","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/model_merger/base_model_merger.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/model_merger/base_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8896f4e2bafe8abc"}},{"code_sha256_prefix":"94209884dcb2308e","entry":"get_dynamic_pipeline_shards","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/model_merger/megatron_model_merger.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/model_merger/megatron_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94209884dcb2308e"}},{"code_sha256_prefix":"735e93af30b15fa9","entry":"is_ipv4","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/utils/net_utils.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"735e93af30b15fa9"}},{"code_sha256_prefix":"83b1686d7ac03c21","entry":"unpad_dataproto","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/protocol.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83b1686d7ac03c21"}},{"code_sha256_prefix":"319697b76561a1e4","entry":"get_generation_config","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"319697b76561a1e4"}},{"code_sha256_prefix":"1ac72f97fbbef4f8","entry":"get_huggingface_actor_config","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ac72f97fbbef4f8"}},{"code_sha256_prefix":"acf240079d434746","entry":"get_weight_loader","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acf240079d434746"}},{"code_sha256_prefix":"2aeadc44472f2c85","entry":"get_weight_saver","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2aeadc44472f2c85"}},{"code_sha256_prefix":"705afe175ad8e404","entry":"is_ipv6","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/utils/net_utils.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"705afe175ad8e404"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"Qwen-Applications/CLIPO","repo_kind":"found_in_text","path":"verl/utils/model.py","file_url":"https://github.com/Qwen-Applications/CLIPO/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}