{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tlcr-token-level-continuous-reward-for-fine","title":"TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback","arxiv_id":"2407.16574","date":"2024-07-23","proceeding":null,"authors":["Eunseop Yoon","Hee Suk Yoon","SooHwan Eom","Gunsoo Han","Daniel Wontae Nam","DaeJin Jo","Kyoung-Woon On","Mark A. Hasegawa-Johnson","Sungwoong Kim","Chang D. Yoo"],"abstract":"Reinforcement Learning from Human Feedback (RLHF) leverages human preference data to train language models to align more closely with human essence. These human preference data, however, are labeled at the sequence level, creating a mismatch between sequence-level preference labels and tokens, which are autoregressively generated from the language model. Although several recent approaches have tried to provide token-level (i.e., dense) rewards for each individual token, these typically rely on predefined discrete reward values (e.g., positive: +1, negative: -1, neutral: 0), failing to account for varying degrees of preference inherent to each token. To address this limitation, we introduce TLCR (Token-Level Continuous Reward) for RLHF, which incorporates a discriminator trained to distinguish positive and negative tokens, and the confidence of the discriminator is used to assign continuous rewards to each token considering the context. Extensive experiments show that our proposed TLCR leads to consistent performance improvements over previous sequence-level or token-level discrete rewards on open-ended generation benchmarks.","url_abs":"https://arxiv.org/abs/2407.16574v2","url_pdf":"https://arxiv.org/pdf/2407.16574v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tlcr-token-level-continuous-reward-for-fine","repo_url":"https://github.com/esyoon7/rlhf-tlcr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.16574","atlas_url":"https://app.syntology.ai/?focus=2407.16574","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16574"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/esyoon7/rlhf-tlcr","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":6,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"0fe82a947dc39b42","entry":"apply_rotary_pos_emb","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/model/discriminator_uni_head_reward_model.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/model/discriminator_uni_head_reward_model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe82a947dc39b42"}},{"code_sha256_prefix":"70fbf92c5d0e9276","entry":"extract_errors","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"dataset/get_token_distance.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/dataset/get_token_distance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70fbf92c5d0e9276"}},{"code_sha256_prefix":"2f0a690c70d11c23","entry":"gather_log_probs","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"step3_rlhf_finetuning_tlcr/ppo_trainer.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/step3_rlhf_finetuning_tlcr/ppo_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f0a690c70d11c23"}},{"code_sha256_prefix":"e221a76f7d7a3d73","entry":"get_eval_ds_config","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/ds_utils.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/ds_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e221a76f7d7a3d73"}},{"code_sha256_prefix":"e826f09e32e17ac5","entry":"levenshtein_alignment","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"dataset/get_token_distance.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/dataset/get_token_distance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e826f09e32e17ac5"}},{"code_sha256_prefix":"e617041853afd002","entry":"only_optimize_lora_parameters","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/module/lora.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/module/lora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e617041853afd002"}},{"code_sha256_prefix":"dc7e4cf018de4ad4","entry":"process_errors","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"dataset/get_token_distance.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/dataset/get_token_distance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc7e4cf018de4ad4"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/model/reward_model.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/model/reward_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"c85065a94b42d887","entry":"softmax_cross_entropy_with_softtarget","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/model/discriminator_uni_head_reward_model.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/model/discriminator_uni_head_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c85065a94b42d887"}},{"code_sha256_prefix":"429d42a3bf925da5","entry":"log_init","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"step3_rlhf_finetuning_tlcr/rlhf_engine.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/step3_rlhf_finetuning_tlcr/rlhf_engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"429d42a3bf925da5"}},{"code_sha256_prefix":"0b516564e07367e0","entry":"new_forward","repo":"esyoon7/rlhf-tlcr","repo_kind":"official","path":"utils/model/discriminator_uni_head_reward_model.py","file_url":"https://github.com/esyoon7/rlhf-tlcr/blob/HEAD/utils/model/discriminator_uni_head_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b516564e07367e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}