{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/earlier-tokens-contribute-more-learning","title":"Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective","arxiv_id":"2502.14340","date":"2025-02-20","proceeding":null,"authors":["Ruichen Shao","Bei Li","Gangao Liu","Yang Chen","Xiang Zhou","Jingang Wang","Xunliang Cai","Peng Li"],"abstract":"Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at \\url{https://github.com/LotuSrc/D2PO}.","url_abs":"https://arxiv.org/abs/2502.14340v1","url_pdf":"https://arxiv.org/pdf/2502.14340v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"earlier-tokens-contribute-more-learning","repo_url":"https://github.com/lotusrc/d2po","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dpo","method_name":"DPO"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.14340","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.14340"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LotuSrc/D2PO","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lotusrc/d2po","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dbea5358be6b5ad0","entry":"create_score_evaluation_response","repo":"LotuSrc/D2PO","repo_kind":"official","path":"src/llamafactory/api/chat.py","file_url":"https://github.com/LotuSrc/D2PO/blob/HEAD/src/llamafactory/api/chat.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbea5358be6b5ad0"}},{"code_sha256_prefix":"52a9ebcf2e9dee87","entry":"dictify","repo":"LotuSrc/D2PO","repo_kind":"official","path":"src/llamafactory/api/common.py","file_url":"https://github.com/LotuSrc/D2PO/blob/HEAD/src/llamafactory/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52a9ebcf2e9dee87"}},{"code_sha256_prefix":"57583cf880d95444","entry":"jsonify","repo":"LotuSrc/D2PO","repo_kind":"official","path":"src/llamafactory/api/common.py","file_url":"https://github.com/LotuSrc/D2PO/blob/HEAD/src/llamafactory/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57583cf880d95444"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}