{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/preference-grounded-token-level-guidance-for-1","title":"Preference-grounded Token-level Guidance for Language Model Fine-tuning","arxiv_id":"2306.00398","date":"2023-06-01","proceeding":"NeurIPS 2023 11","authors":["Shentao Yang","Shujian Zhang","Congying Xia","Yihao Feng","Caiming Xiong","Mingyuan Zhou"],"abstract":"Aligning language models (LMs) with preferences is an important problem in natural language generation. A key challenge is that preferences are typically provided at the sequence level while LM training and generation both occur at the token level. There is, therefore, a granularity mismatch between the preference and the LM training losses, which may complicate the learning problem. In this paper, we address this issue by developing an alternate training process, where we iterate between grounding the sequence-level preference into token-level training guidance, and improving the LM with the learned guidance. For guidance learning, we design a framework that extends the pairwise-preference learning in imitation learning to both variable-length LM generation and the utilization of the preference among multiple generations. For LM training, based on the amount of supervised data, we present two minimalist learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks -- discrete-prompt generation and text summarization.","url_abs":"https://arxiv.org/abs/2306.00398v3","url_pdf":"https://arxiv.org/pdf/2306.00398v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"preference-grounded-token-level-guidance-for-1","repo_url":"https://github.com/shentao-yang/preference_grounded_guidance","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"preference-grounded-token-level-guidance-for-1","repo_url":"https://github.com/yinyueqin/denserewardrlhf-ppo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.00398","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00398"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Shentao-YANG/Preference_Grounded_Guidance","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shentao-yang/preference_grounded_guidance","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yinyueqin/denserewardrlhf-ppo","reach":null}],"summary":{"ran":2,"ran_draft_wrong":6,"unverified":10},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1},"listed":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"5017ac033cf345d5","entry":"Experience","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5017ac033cf345d5"}},{"code_sha256_prefix":"6fdf28a6230ef77d","entry":"RewardModel","repo":"shentao-yang/preference_grounded_guidance","repo_kind":"official","path":"sum_task/reward_model.py","file_url":"https://github.com/shentao-yang/preference_grounded_guidance/blob/HEAD/sum_task/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fdf28a6230ef77d"}},{"code_sha256_prefix":"e97b4f8302932235","entry":"_build_one_layer_mlp","repo":"shentao-yang/preference_grounded_guidance","repo_kind":"official","path":"sum_task/reward_model.py","file_url":"https://github.com/shentao-yang/preference_grounded_guidance/blob/HEAD/sum_task/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e97b4f8302932235"}},{"code_sha256_prefix":"8572f73b51db94c4","entry":"compute_approx_kl","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8572f73b51db94c4"}},{"code_sha256_prefix":"959df7824bf09cc1","entry":"log_probs_from_logits","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"959df7824bf09cc1"}},{"code_sha256_prefix":"bff9c1af0e65b23b","entry":"masked_mean","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bff9c1af0e65b23b"}},{"code_sha256_prefix":"3e5662344a15cfae","entry":"reset_position_ids","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e5662344a15cfae"}},{"code_sha256_prefix":"938b6f55326552be","entry":"to","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"938b6f55326552be"}},{"code_sha256_prefix":"2c1a919b8596c263","entry":"Actor","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c1a919b8596c263"}},{"code_sha256_prefix":"832009e99448c415","entry":"NaiveExperienceMaker","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"832009e99448c415"}},{"code_sha256_prefix":"8f83f8e37a1fc568","entry":"_init_weights","repo":"shentao-yang/preference_grounded_guidance","repo_kind":"official","path":"sum_task/reward_model.py","file_url":"https://github.com/shentao-yang/preference_grounded_guidance/blob/HEAD/sum_task/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f83f8e37a1fc568"}},{"code_sha256_prefix":"b96e5125bdaa4277","entry":"compute_reward","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b96e5125bdaa4277"}},{"code_sha256_prefix":"340cb12bdef03f95","entry":"convert_ring_attn_params","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"340cb12bdef03f95"}},{"code_sha256_prefix":"98181e9dfff64330","entry":"pin_memory","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98181e9dfff64330"}},{"code_sha256_prefix":"891ce03e776aef13","entry":"prints","repo":"shentao-yang/preference_grounded_guidance","repo_kind":"official","path":"sum_task/reward_model.py","file_url":"https://github.com/shentao-yang/preference_grounded_guidance/blob/HEAD/sum_task/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"891ce03e776aef13"}},{"code_sha256_prefix":"e93117d99aa6b210","entry":"reset_ring_attn_position_ids","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e93117d99aa6b210"}},{"code_sha256_prefix":"e930cb6424d6bb2d","entry":"retry","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e930cb6424d6bb2d"}},{"code_sha256_prefix":"74b9172d4d4d6e9c","entry":"update_ring_attn_params","repo":"yinyueqin/denserewardrlhf-ppo","repo_kind":"listed","path":"denserlhf/trainer/ppo_utils/experience_maker.py","file_url":"https://github.com/yinyueqin/denserewardrlhf-ppo/blob/HEAD/denserlhf/trainer/ppo_utils/experience_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74b9172d4d4d6e9c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}