{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-02495","title":"Reward-free Alignment for Conflicting Objectives","arxiv_id":"2602.02495","date":"2026-02-02","proceeding":"ICML","authors":["Peter Chen","Xiaopeng Li","Xi Chen","Tianyi Lin"],"abstract":"Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.","url_abs":"https://arxiv.org/abs/2602.02495","url_pdf":"https://arxiv.org/pdf/2602.02495","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.02495","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.02495"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/PeterLauLukChen/RACO","reach":null}],"summary":{"unverified":16},"by_repo_kind":{"found_in_text":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2557e245c7e798c3","entry":"beaver_conversation","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_beaver_reward_cost.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_beaver_reward_cost.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2557e245c7e798c3"}},{"code_sha256_prefix":"8a99d20ff14e3711","entry":"build_text","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_reward_summarization.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_reward_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a99d20ff14e3711"}},{"code_sha256_prefix":"9ebf3925d706325e","entry":"compute_variance","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/calc_raco_metric_stats.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/calc_raco_metric_stats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ebf3925d706325e"}},{"code_sha256_prefix":"681958bea02af0cf","entry":"discover_inputs","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/calc_raco_metric_stats.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/calc_raco_metric_stats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"681958bea02af0cf"}},{"code_sha256_prefix":"b7256244e37b6cb1","entry":"extract_prompt","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/build_summary_eval_prompts.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/build_summary_eval_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7256244e37b6cb1"}},{"code_sha256_prefix":"9072e64887215a81","entry":"main","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/new_score.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/new_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9072e64887215a81"}},{"code_sha256_prefix":"78304958ed3d9382","entry":"main","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/new_score_beaver.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/new_score_beaver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78304958ed3d9382"}},{"code_sha256_prefix":"0591481280db58e2","entry":"main","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/vllm_generate_and_score.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/vllm_generate_and_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0591481280db58e2"}},{"code_sha256_prefix":"6c574acf7317667e","entry":"make_async_client","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/vllm_generate.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/vllm_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c574acf7317667e"}},{"code_sha256_prefix":"fda35ae84a267de1","entry":"make_client","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/vllm_generate.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/vllm_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fda35ae84a267de1"}},{"code_sha256_prefix":"792996f18741b2bb","entry":"parse_logged_dict","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/calc_raco_metric_stats.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/calc_raco_metric_stats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"792996f18741b2bb"}},{"code_sha256_prefix":"c7af484423746243","entry":"read_prompts_from_parquet","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"eval/vllm_generate.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/eval/vllm_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7af484423746243"}},{"code_sha256_prefix":"69c7ec23046c9eff","entry":"score_pairs","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_faithful_summarization.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_faithful_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69c7ec23046c9eff"}},{"code_sha256_prefix":"b7550e6553b4157c","entry":"score_texts","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_reward_summarization.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_reward_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7550e6553b4157c"}},{"code_sha256_prefix":"ee4d9f7f86635181","entry":"strip_system_prompt","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_faithful_summarization.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_faithful_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee4d9f7f86635181"}},{"code_sha256_prefix":"9197f51c0cb72bcc","entry":"strip_system_prompt","repo":"PeterLauLukChen/RACO","repo_kind":"found_in_text","path":"script/score_reward_summarization.py","file_url":"https://github.com/PeterLauLukChen/RACO/blob/HEAD/script/score_reward_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9197f51c0cb72bcc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}