{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gram-a-generative-foundation-reward-model-for","title":"GRAM: A Generative Foundation Reward Model for Reward Generalization","arxiv_id":"2506.14175","date":"2025-06-17","proceeding":null,"authors":["Chenglong Wang","Yang Gan","Yifu Huo","Yongyu Mu","Qiaozhi He","Murun Yang","Bei Li","Tong Xiao","Chunliang Zhang","Tongran Liu","Jingbo Zhu"],"abstract":"In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data. In this paper, we explore methods that train reward models using both unlabeled and labeled data. Building on the generative models in LLMs, we develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss. This result, in turn, provides a new view of training reward models, which links generative models and discriminative models under the same class of training objectives. The outcome of these techniques is a foundation reward model, which can be applied to a wide range of tasks with little or no further fine-tuning effort. Extensive experiments show that this model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.","url_abs":"https://arxiv.org/abs/2506.14175v2","url_pdf":"https://arxiv.org/pdf/2506.14175v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gram-a-generative-foundation-reward-model-for","repo_url":"https://github.com/niutrans/gram","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gram-a-generative-foundation-reward-model-for","repo_url":"https://github.com/wangclnlp/GRAM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.14175","atlas_url":"https://app.syntology.ai/?focus=2506.14175","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.14175"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/NiuTrans/GRAM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/niutrans/gram","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangclnlp/GRAM","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9ec4e052e8733047","entry":"create_client","repo":"NiuTrans/GRAM","repo_kind":"official","path":"extensions/GRAM-RR/call_vllm_with_multi_port.py","file_url":"https://github.com/NiuTrans/GRAM/blob/HEAD/extensions/GRAM-RR/call_vllm_with_multi_port.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ec4e052e8733047"}},{"code_sha256_prefix":"d37a8e250f230213","entry":"load_convert_data","repo":"NiuTrans/GRAM","repo_kind":"official","path":"extensions/GRAM-RR/call_vllm_with_multi_port.py","file_url":"https://github.com/NiuTrans/GRAM/blob/HEAD/extensions/GRAM-RR/call_vllm_with_multi_port.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d37a8e250f230213"}},{"code_sha256_prefix":"9cc05b8398aba7fa","entry":"load_existing_ids","repo":"NiuTrans/GRAM","repo_kind":"official","path":"extensions/GRAM-RR/call_vllm_with_multi_port.py","file_url":"https://github.com/NiuTrans/GRAM/blob/HEAD/extensions/GRAM-RR/call_vllm_with_multi_port.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc05b8398aba7fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}