{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-25757","title":"OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning","arxiv_id":"2606.25757","date":"2026-06-24","proceeding":null,"authors":["Wenxuan Jiang","Zining Fan","Zijian Zhang","Xuecheng Wu","Hongming Tan","Haoyang Dai","Xiaoyu Li","Xuezhi Cao","Ninghao Liu"],"abstract":"Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.","url_abs":"https://arxiv.org/abs/2606.25757","url_pdf":"https://arxiv.org/pdf/2606.25757","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.25757","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.25757"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/pangpang-xuan/OPERA","reach":null}],"summary":{"ran":11},"by_repo_kind":{"found_in_text":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"6b3a32889f385ce0","entry":"entropy_from_logprobs","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"sft_data_sync/utils.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/sft_data_sync/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b3a32889f385ce0"}},{"code_sha256_prefix":"9c496dae615ba2a8","entry":"generate_config_from_args","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/verl/model_merger/base_model_merger.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/verl/model_merger/base_model_merger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c496dae615ba2a8"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"be670aa6ccf31828","entry":"merge_by_placement","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/all2hf.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/all2hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be670aa6ccf31828"}},{"code_sha256_prefix":"c30b2a96103abd2c","entry":"reward_func","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"verl/recipe/r1/reward_score.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/verl/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c30b2a96103abd2c"}},{"code_sha256_prefix":"59527734e1a3239f","entry":"vote_outputs_unwrap","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"sft_data_sync/utils.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/sft_data_sync/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59527734e1a3239f"}},{"code_sha256_prefix":"d23ae2928c8a88f4","entry":"vote_prompt_wrap","repo":"pangpang-xuan/OPERA","repo_kind":"found_in_text","path":"sft_data_sync/utils.py","file_url":"https://github.com/pangpang-xuan/OPERA/blob/HEAD/sft_data_sync/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d23ae2928c8a88f4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}