{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/getting-more-juice-out-of-the-sft-data-reward","title":"Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment","arxiv_id":"2405.17888","date":"2024-05-28","proceeding":null,"authors":["Jiaxiang Li","Siliang Zeng","Hoi-To Wai","Chenliang Li","Alfredo Garcia","Mingyi Hong"],"abstract":"Aligning human preference and value is an important requirement for contemporary foundation models. State-of-the-art techniques such as Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine-tuning (SFT), where the model is fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn a reward model, which is in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as a proxy to human preference, and it is critical to guide the RL step towards improving the model quality. In this work, we argue that the SFT stage significantly benefits from learning a reward model as well. Instead of using the human demonstration data directly via supervised learning, we propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. This approach leads to new SFT algorithms that are not only efficient to implement, but are robust to the presence of low-quality supervised learning data. Moreover, we discover a connection between the proposed IRL based approach, and a recent line of works called Self-Play Fine-tune (SPIN). Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align 1B and 7B models using proposed methods and evaluate them on a reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.","url_abs":"https://arxiv.org/abs/2405.17888v3","url_pdf":"https://arxiv.org/pdf/2405.17888v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"getting-more-juice-out-of-the-sft-data-reward","repo_url":"https://github.com/jasonjiaxiangli/reward_learning_sft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.17888","atlas_url":"https://app.syntology.ai/?focus=2405.17888","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17888"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jasonjiaxiangli/reward_learning_sft","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":2,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"12f6803880f9b17e","entry":"extract_anthropic_prompt","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/generate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12f6803880f9b17e"}},{"code_sha256_prefix":"6c6b96dbc3ac9f5a","entry":"prepare_prompts","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c6b96dbc3ac9f5a"}},{"code_sha256_prefix":"1eb55970dd4151b3","entry":"prepare_prompts","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1eb55970dd4151b3"}},{"code_sha256_prefix":"561efc3deb84664e","entry":"create_comparison_dataset","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/reward_learning_AIHF.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/reward_learning_AIHF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"561efc3deb84664e"}},{"code_sha256_prefix":"4ddaf7ff38b5dd2c","entry":"create_expert_dataset","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/reward_learning_AIHF.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/reward_learning_AIHF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ddaf7ff38b5dd2c"}},{"code_sha256_prefix":"2e75dde84c46b293","entry":"get_arc","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e75dde84c46b293"}},{"code_sha256_prefix":"da9d319c58d46ad0","entry":"get_hh","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/generate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da9d319c58d46ad0"}},{"code_sha256_prefix":"e9fd32097c8c06b6","entry":"get_wino","repo":"jasonjiaxiangli/reward_learning_sft","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/jasonjiaxiangli/reward_learning_sft/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9fd32097c8c06b6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}