{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-24040","title":"RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models","arxiv_id":"2602.24040","date":"2026-02-27","proceeding":null,"authors":["Daniel Yang","Samuel Stante","Florian Redhardt","Lena Libon","Parnian Kassraie","Ido Hakimi","Barna Pásztor","Andreas Krause"],"abstract":"Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.","url_abs":"https://arxiv.org/abs/2602.24040","url_pdf":"https://arxiv.org/pdf/2602.24040","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.24040","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.24040"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/lasgroup/rewarduq","reach":null}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f556c040de75918a","entry":"compute_pref_probs","repo":"lasgroup/rewarduq","repo_kind":"found_in_text","path":"src/rewarduq/metrics.py","file_url":"https://github.com/lasgroup/rewarduq/blob/HEAD/src/rewarduq/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f556c040de75918a"}},{"code_sha256_prefix":"843f65772f1f2ff5","entry":"compute_reward_margins","repo":"lasgroup/rewarduq","repo_kind":"found_in_text","path":"src/rewarduq/metrics.py","file_url":"https://github.com/lasgroup/rewarduq/blob/HEAD/src/rewarduq/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"843f65772f1f2ff5"}},{"code_sha256_prefix":"8fdfb3ebc3b567cb","entry":"compute_statistics","repo":"lasgroup/rewarduq","repo_kind":"found_in_text","path":"src/rewarduq/metrics.py","file_url":"https://github.com/lasgroup/rewarduq/blob/HEAD/src/rewarduq/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8fdfb3ebc3b567cb"}},{"code_sha256_prefix":"1bdb717c20c65159","entry":"get_torch_dtype","repo":"lasgroup/rewarduq","repo_kind":"found_in_text","path":"src/rewarduq/utils/models.py","file_url":"https://github.com/lasgroup/rewarduq/blob/HEAD/src/rewarduq/utils/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bdb717c20c65159"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}