{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/toward-evaluative-thinking-meta-policy","title":"Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models","arxiv_id":"2504.20157","date":"2025-04-28","proceeding":null,"authors":["Zae Myung Kim","Chanwoo Park","Vipul Raheja","Suin Kim","Dongyeop Kang"],"abstract":"Reward-based alignment methods for large language models (LLMs) face two key limitations: vulnerability to reward hacking, where models exploit flaws in the reward signal; and reliance on brittle, labor-intensive prompt engineering when LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a framework that addresses these challenges by integrating a meta-reward model that dynamically refines the reward model's prompt throughout training. In MPO, the meta-reward model monitors the evolving training context and continuously adjusts the reward model's prompt to maintain high alignment, providing an adaptive reward signal that resists exploitation by the policy. This meta-learning approach promotes a more stable policy optimization, and greatly reduces the need for manual reward prompt design. It yields performance on par with or better than models guided by extensively hand-crafted reward prompts. Furthermore, we show that MPO maintains its effectiveness across diverse tasks, from essay writing to mathematical reasoning, without requiring specialized reward designs. Beyond standard RLAIF, MPO's meta-learning formulation is readily extensible to higher-level alignment frameworks. Overall, this method addresses theoretical and practical challenges in reward-based RL alignment for LLMs, paving the way for more robust and adaptable alignment strategies. The code and data can be accessed at: https://github.com/minnesotanlp/mpo","url_abs":"https://arxiv.org/abs/2504.20157v2","url_pdf":"https://arxiv.org/pdf/2504.20157v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"toward-evaluative-thinking-meta-policy","repo_url":"https://github.com/minnesotanlp/mpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"rlaif","method_name":"RLAIF"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.20157","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.20157"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/minnesotanlp/mpo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_violates":1,"unverified":11},"by_repo_kind":{"official":{"samples":15,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"dfd0ecd9cb25af69","entry":"flatten_dict","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"dfd0ecd9cb25af69"}},{"code_sha256_prefix":"20063c675d638a77","entry":"is_equiv","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/extras/mpo/rm_math_reasoning.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/extras/mpo/rm_math_reasoning.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"20063c675d638a77"}},{"code_sha256_prefix":"4f7075aac2163cbc","entry":"masked_mean","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4f7075aac2163cbc"}},{"code_sha256_prefix":"67607956afbb2a83","entry":"masked_var","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"67607956afbb2a83"}},{"code_sha256_prefix":"1167d54333f6f7a3","entry":"aesthetic_scorer","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/models/auxiliary_modules.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/models/auxiliary_modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1167d54333f6f7a3"}},{"code_sha256_prefix":"0b7223ebdefef219","entry":"apply_chat_template","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/data_utils.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0b7223ebdefef219"}},{"code_sha256_prefix":"cf11df858b102576","entry":"extract_boxed_content","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/extras/mpo/rm_math_reasoning.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/extras/mpo/rm_math_reasoning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cf11df858b102576"}},{"code_sha256_prefix":"2768846777b28be1","entry":"get_task_specific_gatherings","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/trainer/mpo_trainer.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/trainer/mpo_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2768846777b28be1"}},{"code_sha256_prefix":"61c97cdf90c40f26","entry":"is_conversational","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/data_utils.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"61c97cdf90c40f26"}},{"code_sha256_prefix":"2bb6075cd6b76760","entry":"is_liger_kernel_available","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/import_utils.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/import_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2bb6075cd6b76760"}},{"code_sha256_prefix":"4c2c416f1561edde","entry":"maybe_apply_chat_template","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/data_utils.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4c2c416f1561edde"}},{"code_sha256_prefix":"62a8858c70afef56","entry":"prepare_essay_writing_dataset","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/extras/mpo/mpo_datasets.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/extras/mpo/mpo_datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"62a8858c70afef56"}},{"code_sha256_prefix":"963b371a14432816","entry":"prepare_mathematical_reasoning_dataset","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/extras/mpo/mpo_datasets.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/extras/mpo/mpo_datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"963b371a14432816"}},{"code_sha256_prefix":"9c740b4e69c3670f","entry":"prepare_summarization_dataset","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/extras/mpo/mpo_datasets.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/extras/mpo/mpo_datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9c740b4e69c3670f"}},{"code_sha256_prefix":"03e50437a6e99c4f","entry":"to_serializable","repo":"minnesotanlp/mpo","repo_kind":"official","path":"trl/trainer/mpo_trainer.py","file_url":"https://github.com/minnesotanlp/mpo/blob/HEAD/trl/trainer/mpo_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"03e50437a6e99c4f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}