{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dense-reward-for-free-in-reinforcement","title":"Dense Reward for Free in Reinforcement Learning from Human Feedback","arxiv_id":"2402.00782","date":"2024-02-01","proceeding":null,"authors":["Alex J. Chan","Hao Sun","Samuel Holt","Mihaela van der Schaar"],"abstract":"Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the LLM has to take many \"actions\" (selecting individual tokens) and only receives a single, sparse reward at the end of an episode, a setup that is known to be difficult to optimise in traditional reinforcement learning. In this work we leverage the fact that the reward model contains more information than just its scalar output, in particular, it calculates an attention map over tokens as part of the transformer architecture. We use these attention weights to redistribute the reward along the whole completion, effectively densifying the signal and highlighting the most important tokens, all without incurring extra computational cost or requiring any additional modelling. We demonstrate that, theoretically, this approach is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.","url_abs":"https://arxiv.org/abs/2402.00782v1","url_pdf":"https://arxiv.org/pdf/2402.00782v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dense-reward-for-free-in-reinforcement","repo_url":"https://github.com/xanderjc/attention-based-credit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.00782","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.00782"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xanderjc/attention-based-credit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"656b5c1bcfee332a","entry":"check_special_tokens","repo":"xanderjc/attention-based-credit","repo_kind":"official","path":"abcrl/helpers/tokenizer_equivalence.py","file_url":"https://github.com/xanderjc/attention-based-credit/blob/HEAD/abcrl/helpers/tokenizer_equivalence.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"656b5c1bcfee332a"}},{"code_sha256_prefix":"556ee836f3fe3f6c","entry":"get_attention_distribution","repo":"xanderjc/attention-based-credit","repo_kind":"official","path":"abcrl/attention/redistribution.py","file_url":"https://github.com/xanderjc/attention-based-credit/blob/HEAD/abcrl/attention/redistribution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"556ee836f3fe3f6c"}},{"code_sha256_prefix":"6d632dad98d9d43e","entry":"get_reward","repo":"xanderjc/attention-based-credit","repo_kind":"official","path":"abcrl/reward_modelling/starling.py","file_url":"https://github.com/xanderjc/attention-based-credit/blob/HEAD/abcrl/reward_modelling/starling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d632dad98d9d43e"}},{"code_sha256_prefix":"9bd8f37d391d794d","entry":"build_nectar_dataset","repo":"xanderjc/attention-based-credit","repo_kind":"official","path":"abcrl/datasets/nectar.py","file_url":"https://github.com/xanderjc/attention-based-credit/blob/HEAD/abcrl/datasets/nectar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bd8f37d391d794d"}},{"code_sha256_prefix":"43a9ad195396ee88","entry":"get_generator_attention_distribution","repo":"xanderjc/attention-based-credit","repo_kind":"official","path":"abcrl/attention/redistribution.py","file_url":"https://github.com/xanderjc/attention-based-credit/blob/HEAD/abcrl/attention/redistribution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43a9ad195396ee88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}