{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controllable-preference-optimization-toward","title":"Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment","arxiv_id":"2402.19085","date":"2024-02-29","proceeding":null,"authors":["Yiju Guo","Ganqu Cui","Lifan Yuan","Ning Ding","Zexu Sun","Bowen Sun","Huimin Chen","Ruobing Xie","Jie zhou","Yankai Lin","Zhiyuan Liu","Maosong Sun"],"abstract":"Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the \"alignment tax\" -a compromise where enhancements in alignment within one objective (e.g.,harmlessness) can diminish performance in others (e.g.,helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the \"3H\" (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.","url_abs":"https://arxiv.org/abs/2402.19085v3","url_pdf":"https://arxiv.org/pdf/2402.19085v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controllable-preference-optimization-toward","repo_url":"https://github.com/OpenBMB/CPO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"}],"methods":[],"datasets_introduced":[{"slug":"ultrasafety","name":"UltraSafety","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.19085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.19085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenBMB/CPO","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"73d8c38b16158fdb","entry":"GetRateByKey","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/data_preparation/data_dpo_ultrafeedback.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/data_preparation/data_dpo_ultrafeedback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73d8c38b16158fdb"}},{"code_sha256_prefix":"f4160c311985d81b","entry":"SampleTargetResponses","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/data_preparation/data_dpo_ultrafeedback.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/data_preparation/data_dpo_ultrafeedback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4160c311985d81b"}},{"code_sha256_prefix":"312e0cabadf2c4af","entry":"changeToJson","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/data_preparation/lib/file_func.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/data_preparation/lib/file_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"312e0cabadf2c4af"}},{"code_sha256_prefix":"f87c81eda77d317f","entry":"further_process_data","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/cdpo/cdpo_ultrafeedback.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/cdpo/cdpo_ultrafeedback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f87c81eda77d317f"}},{"code_sha256_prefix":"42b3af088826241e","entry":"further_process_data","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/cdpo/cdpo_ultrasafety.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/cdpo/cdpo_ultrasafety.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42b3af088826241e"}},{"code_sha256_prefix":"fba000175d7bcb8e","entry":"preprocess_data","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/cdpo/cdpo_ultrafeedback.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/cdpo/cdpo_ultrafeedback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fba000175d7bcb8e"}},{"code_sha256_prefix":"97785ad2ada07093","entry":"preprocess_data","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/cdpo/cdpo_ultrasafety.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/cdpo/cdpo_ultrasafety.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97785ad2ada07093"}},{"code_sha256_prefix":"88885e2717e9acca","entry":"readJsonString","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/data_preparation/lib/file_func.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/data_preparation/lib/file_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88885e2717e9acca"}},{"code_sha256_prefix":"4334eae9875cd51c","entry":"readfile","repo":"OpenBMB/CPO","repo_kind":"official","path":"src/CDPO/data_preparation/lib/file_func.py","file_url":"https://github.com/OpenBMB/CPO/blob/HEAD/src/CDPO/data_preparation/lib/file_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4334eae9875cd51c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}