{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/it-takes-two-on-the-seamlessness-between","title":"It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF","arxiv_id":"2406.07971","date":"2024-06-12","proceeding":null,"authors":["Taiming Lu","Lingfeng Shen","Xinyu Yang","Weiting Tan","Beidi Chen","Huaxiu Yao"],"abstract":"Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we propose to examine their interactions during fine-tuning, introducing the concept of seamlessness. Our study starts with observing the saturation phenomenon, where continual improvements in RM and PM do not translate into RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM responses, resulting in a 35% mismatch rate with human preferences, highlighting a significant discrepancy between PM and RM. To measure seamlessness between PM and RM without human effort, we propose an automatic metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments induced by data samples. We validate the effectiveness of SEAM in data selection and model augmentation. Our experiments demonstrate that (1) using SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2) SEAM-guided model augmentation results in a 4% performance improvement over standard augmentation methods.","url_abs":"https://arxiv.org/abs/2406.07971v2","url_pdf":"https://arxiv.org/pdf/2406.07971v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"it-takes-two-on-the-seamlessness-between","repo_url":"https://github.com/taiminglu/seamless","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"seam","method_name":"SEAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.07971","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07971"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taiminglu/seamless","reach":{"status":"ok"}}],"summary":{"ran":11,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":12,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"e1333323e0d954bc","entry":"calculate_log_probability","repo":"taiminglu/seamless","repo_kind":"official","path":"code/SEAM/policy/get_prob.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/SEAM/policy/get_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1333323e0d954bc"}},{"code_sha256_prefix":"fddcce152bc7d3bd","entry":"collator","repo":"taiminglu/seamless","repo_kind":"official","path":"code/train/rl.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/train/rl.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fddcce152bc7d3bd"}},{"code_sha256_prefix":"f8f1cb4c32403020","entry":"combine_and_pad_tokens","repo":"taiminglu/seamless","repo_kind":"official","path":"code/SEAM/policy/get_prob.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/SEAM/policy/get_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8f1cb4c32403020"}},{"code_sha256_prefix":"31218514d492527d","entry":"encode_and_search","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/contrast/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/contrast/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31218514d492527d"}},{"code_sha256_prefix":"fb97e66f814910e9","entry":"get_completion","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/gpt/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/gpt/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb97e66f814910e9"}},{"code_sha256_prefix":"845ccc17a73fe529","entry":"get_prompt","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/gpt/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/gpt/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"845ccc17a73fe529"}},{"code_sha256_prefix":"48fc27d8ef2d0595","entry":"get_results","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/adv/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/adv/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48fc27d8ef2d0595"}},{"code_sha256_prefix":"646f36a115dc20f4","entry":"get_reward","repo":"taiminglu/seamless","repo_kind":"official","path":"code/SEAM/reward/get_reward.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/SEAM/reward/get_reward.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"646f36a115dc20f4"}},{"code_sha256_prefix":"55737ec700776650","entry":"get_text","repo":"taiminglu/seamless","repo_kind":"official","path":"code/SEAM/reward/get_reward.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/SEAM/reward/get_reward.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55737ec700776650"}},{"code_sha256_prefix":"43b18b0788bb9a75","entry":"load_data","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/contrast/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/contrast/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43b18b0788bb9a75"}},{"code_sha256_prefix":"14d77a6c7cf50243","entry":"load_data","repo":"taiminglu/seamless","repo_kind":"official","path":"code/retrival/gpt/retrival.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/retrival/gpt/retrival.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14d77a6c7cf50243"}},{"code_sha256_prefix":"148dc2c38b7cd922","entry":"prepare_sample_text","repo":"taiminglu/seamless","repo_kind":"official","path":"code/train/sft.py","file_url":"https://github.com/taiminglu/seamless/blob/HEAD/code/train/sft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"148dc2c38b7cd922"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}