{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bpo-supercharging-online-preference-learning","title":"BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment","arxiv_id":"2406.12168","date":"2024-06-18","proceeding":null,"authors":["Wenda Xu","Jiachen Li","William Yang Wang","Lei LI"],"abstract":"Direct alignment from preferences (DAP) has emerged as a promising paradigm for aligning large language models (LLMs) to human desiderata from pre-collected, offline preference datasets. While recent studies indicate that existing offline DAP methods can directly benefit from online training samples, we highlight the need to develop specific online DAP algorithms to fully harness the power of online training. Specifically, we identify that the learned LLM should adhere to the proximity of the behavior LLM, which collects the training samples. To this end, we propose online Preference Optimization in proximity to the Behavior LLM (BPO), emphasizing the importance of constructing a proper trust region for LLM alignment. We conduct extensive experiments to validate the effectiveness and applicability of our approach by integrating it with various DAP methods, resulting in significant performance improvements across a wide range of tasks when training with the same amount of preference data. Even when only introducing one additional data collection phase, our online BPO improves its offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on Anthropic Helpfulness in terms of win rate against human reference text.","url_abs":"https://arxiv.org/abs/2406.12168v4","url_pdf":"https://arxiv.org/pdf/2406.12168v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bpo-supercharging-online-preference-learning","repo_url":"https://github.com/xu1998hz/bpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12168","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12168"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xu1998hz/bpo","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"798f579e92daa2f0","entry":"completions_with_backoff_openai","repo":"xu1998hz/bpo","repo_kind":"official","path":"eval/benchmark_llm.py","file_url":"https://github.com/xu1998hz/bpo/blob/HEAD/eval/benchmark_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"798f579e92daa2f0"}},{"code_sha256_prefix":"7c3a4905aae72fe6","entry":"completions_with_backoff_openai","repo":"xu1998hz/bpo","repo_kind":"official","path":"train/continue_lora_train.py","file_url":"https://github.com/xu1998hz/bpo/blob/HEAD/train/continue_lora_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c3a4905aae72fe6"}},{"code_sha256_prefix":"d763030923ad8b06","entry":"get_first_batch_data","repo":"xu1998hz/bpo","repo_kind":"official","path":"train/debug_loss_converge.py","file_url":"https://github.com/xu1998hz/bpo/blob/HEAD/train/debug_loss_converge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d763030923ad8b06"}},{"code_sha256_prefix":"c6c9a243fd55cc86","entry":"set_up_single_lora","repo":"xu1998hz/bpo","repo_kind":"official","path":"train/continue_lora_train.py","file_url":"https://github.com/xu1998hz/bpo/blob/HEAD/train/continue_lora_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6c9a243fd55cc86"}},{"code_sha256_prefix":"81c34807e588c39a","entry":"logits_compute","repo":"xu1998hz/bpo","repo_kind":"official","path":"inference/reward.py","file_url":"https://github.com/xu1998hz/bpo/blob/HEAD/inference/reward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81c34807e588c39a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}