{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/personalized-language-modeling-from","title":"Personalized Language Modeling from Personalized Human Feedback","arxiv_id":"2402.05133","date":"2024-02-06","proceeding":null,"authors":["Xinyu Li","Ruiyang Zhou","Zachary C. Lipton","Liu Leqi"],"abstract":"Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.","url_abs":"https://arxiv.org/abs/2402.05133v3","url_pdf":"https://arxiv.org/pdf/2402.05133v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"personalized-language-modeling-from","repo_url":"https://github.com/humainlab/personalized_rlhf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"preference-mapping","task_name":"Preference Mapping"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05133","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05133"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/humainlab/personalized_rlhf","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"961fdf789fac9510","entry":"build_tldr_dataset_dpo_synthetic","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"prlhf/utils.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/prlhf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"961fdf789fac9510"}},{"code_sha256_prefix":"cca45bb9a111ab77","entry":"compute_reward_modeling_metrics","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/reward_modeling_trainer.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/reward_modeling_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cca45bb9a111ab77"}},{"code_sha256_prefix":"70d6b8109b3f3478","entry":"encode_selected_users","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"prlhf/utils.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/prlhf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70d6b8109b3f3478"}},{"code_sha256_prefix":"f5ff77db7757bc92","entry":"flatten_dict","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/common.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5ff77db7757bc92"}},{"code_sha256_prefix":"46af3eb303562401","entry":"format_output","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/data_preprocessor.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/data_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46af3eb303562401"}},{"code_sha256_prefix":"a71a4bdc37586fe2","entry":"format_prompt","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/data_preprocessor.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/data_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a71a4bdc37586fe2"}},{"code_sha256_prefix":"8b1ea4142642d361","entry":"format_prompt_with_data_frame","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/data_preprocessor.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/data_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b1ea4142642d361"}},{"code_sha256_prefix":"9a1871dd5c967ac1","entry":"let_model_save_mem_when_zero_grad","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"evaluate/alpaca_farm/common.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/evaluate/alpaca_farm/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a1871dd5c967ac1"}},{"code_sha256_prefix":"625d1b68353ab5c4","entry":"load_openai_comparisons","repo":"humainlab/personalized_rlhf","repo_kind":"official","path":"prlhf/utils.py","file_url":"https://github.com/humainlab/personalized_rlhf/blob/HEAD/prlhf/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"625d1b68353ab5c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}