{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rewards-in-context-multi-objective-alignment","title":"Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment","arxiv_id":"2402.10207","date":"2024-02-15","proceeding":null,"authors":["Rui Yang","Xiaoman Pan","Feng Luo","Shuang Qiu","Han Zhong","Dong Yu","Jianshu Chen"],"abstract":"We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around 10% GPU hours compared with multi-objective RL baseline.","url_abs":"https://arxiv.org/abs/2402.10207v6","url_pdf":"https://arxiv.org/pdf/2402.10207v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rewards-in-context-multi-objective-alignment","repo_url":"https://github.com/yangrui2015/ric","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"rewards-in-context-multi-objective-alignment","repo_url":"https://github.com/srzer/mod","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10207","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10207"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srzer/mod","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangrui2015/ric","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1},"listed":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"6ae81097a2ef6cb3","entry":"check_lora_in_model_path","repo":"srzer/mod","repo_kind":"listed","path":"experiment-PPO/ric/multi_reward_models.py","file_url":"https://github.com/srzer/mod/blob/HEAD/experiment-PPO/ric/multi_reward_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ae81097a2ef6cb3"}},{"code_sha256_prefix":"28d5e50dbbf83280","entry":"convert_binary_pil","repo":"yangrui2015/ric","repo_kind":"official","path":"ric_txt2img/image_metric.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric_txt2img/image_metric.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28d5e50dbbf83280"}},{"code_sha256_prefix":"5dc2d5b477d36a34","entry":"encode_jpeg","repo":"yangrui2015/ric","repo_kind":"official","path":"ric_txt2img/image_metric.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric_txt2img/image_metric.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5dc2d5b477d36a34"}},{"code_sha256_prefix":"939f2e01fbf8d40d","entry":"find_pareto_points","repo":"yangrui2015/ric","repo_kind":"official","path":"ric_txt2img/visualize_pareto.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric_txt2img/visualize_pareto.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"939f2e01fbf8d40d"}},{"code_sha256_prefix":"378dcc6d47672a1a","entry":"RewardModels","repo":"srzer/mod","repo_kind":"listed","path":"experiment-PPO/ric/multi_reward_models.py","file_url":"https://github.com/srzer/mod/blob/HEAD/experiment-PPO/ric/multi_reward_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"378dcc6d47672a1a"}},{"code_sha256_prefix":"406b59a243695a5f","entry":"build_dataset_with_preference","repo":"yangrui2015/ric","repo_kind":"official","path":"ric/utils.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"406b59a243695a5f"}},{"code_sha256_prefix":"c2e52a32dd1f6431","entry":"build_dataset_with_preference_n","repo":"yangrui2015/ric","repo_kind":"official","path":"ric/utils.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2e52a32dd1f6431"}},{"code_sha256_prefix":"384dbf3efb62957c","entry":"build_summary_dataset_with_preference_n","repo":"yangrui2015/ric","repo_kind":"official","path":"ric/utils.py","file_url":"https://github.com/yangrui2015/ric/blob/HEAD/ric/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"384dbf3efb62957c"}},{"code_sha256_prefix":"270fd803f1ce23ec","entry":"get_rewards","repo":"srzer/mod","repo_kind":"listed","path":"experiment-PPO/ric/multi_reward_models.py","file_url":"https://github.com/srzer/mod/blob/HEAD/experiment-PPO/ric/multi_reward_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"270fd803f1ce23ec"}},{"code_sha256_prefix":"4f8a90a0fc59d399","entry":"load_reward_model","repo":"srzer/mod","repo_kind":"listed","path":"experiment-PPO/ric/multi_reward_models.py","file_url":"https://github.com/srzer/mod/blob/HEAD/experiment-PPO/ric/multi_reward_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8a90a0fc59d399"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}