{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-20369","title":"Ask a Strong LLM Judge when Your Reward Model is Uncertain","arxiv_id":"2510.20369","date":"2025-10-23","proceeding":"NeurIPS","authors":["Zhenghao Xu","Qin Lu","Qingru Zhang","Liang Qiu","Ilgee Hong","Changlong Yu","Wenlin Yao","Yao Liu","Haoming Jiang","Lihong Li","Hyokun Yun","Tuo Zhao"],"abstract":"Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.","url_abs":"https://arxiv.org/abs/2510.20369","url_pdf":"https://arxiv.org/pdf/2510.20369","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.20369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.20369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/zhenghaoxu-gatech/uncertainty-router","reach":{"status":"ok"}}],"summary":{"unverified":15},"by_repo_kind":{"found_in_text":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"8089f3dec6f94f3d","entry":"create_calibration_plot","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rewardbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rewardbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8089f3dec6f94f3d"}},{"code_sha256_prefix":"7fbf4adeed2e6683","entry":"create_calibration_plot","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rmbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rmbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fbf4adeed2e6683"}},{"code_sha256_prefix":"2c9a27186ef5b823","entry":"create_generative_format","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/hs3.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/hs3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c9a27186ef5b823"}},{"code_sha256_prefix":"2ac57b62a9cd3d26","entry":"create_message_format","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/hs3.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/hs3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ac57b62a9cd3d26"}},{"code_sha256_prefix":"1acad12079a57e2a","entry":"extract_label","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rewardbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rewardbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1acad12079a57e2a"}},{"code_sha256_prefix":"6fd4bef84c9874a0","entry":"generate_symmetric_coordinates","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/openrlhf/cli/serve_pm_router.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/openrlhf/cli/serve_pm_router.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fd4bef84c9874a0"}},{"code_sha256_prefix":"06d156b26b9e4045","entry":"get_deepseek_r1_response","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rewardbench_random.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rewardbench_random.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06d156b26b9e4045"}},{"code_sha256_prefix":"2210b57f6ae065bc","entry":"get_deepseek_r1_response","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/autoif1.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/autoif1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2210b57f6ae065bc"}},{"code_sha256_prefix":"40a49f2ea902d2c8","entry":"plot_accuracy_vs_uncertainty","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rewardbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rewardbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40a49f2ea902d2c8"}},{"code_sha256_prefix":"34caa6c01c04d0b3","entry":"plot_accuracy_vs_uncertainty","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rewardbench_random.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rewardbench_random.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34caa6c01c04d0b3"}},{"code_sha256_prefix":"9e655384960ecbf8","entry":"plot_accuracy_vs_uncertainty","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rmbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rmbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e655384960ecbf8"}},{"code_sha256_prefix":"3e1d6c37c0d661d7","entry":"plot_accuracy_vs_uncertainty","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rmbench_random.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rmbench_random.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e1d6c37c0d661d7"}},{"code_sha256_prefix":"dc1400a32a7e401b","entry":"plot_score_distributions","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/eval_pm_router/eval_rmbench.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/eval_pm_router/eval_rmbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc1400a32a7e401b"}},{"code_sha256_prefix":"412bf7c94e006a38","entry":"plot_strength_histogram","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/src/hs3.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/src/hs3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"412bf7c94e006a38"}},{"code_sha256_prefix":"4081a5e23d9bc90f","entry":"strip_sequence","repo":"zhenghaoxu-gatech/uncertainty-router","repo_kind":"found_in_text","path":"ppo/openrlhf/cli/serve_pm_router.py","file_url":"https://github.com/zhenghaoxu-gatech/uncertainty-router/blob/HEAD/ppo/openrlhf/cli/serve_pm_router.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4081a5e23d9bc90f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}