{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-diverse-preferences-for-large-language","title":"On Diversified Preferences of Large Language Model Alignment","arxiv_id":"2312.07401","date":"2023-12-12","proceeding":null,"authors":["Dun Zeng","Yong Dai","Pengyu Cheng","Longyue Wang","Tianhao Hu","Wanshun Chen","Nan Du","Zenglin Xu"],"abstract":"Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.","url_abs":"https://arxiv.org/abs/2312.07401v5","url_pdf":"https://arxiv.org/pdf/2312.07401v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-diverse-preferences-for-large-language","repo_url":"https://github.com/dunzeng/more","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.07401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.07401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dunzeng/more","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"520e1c7389395ede","entry":"get_data_iter","repo":"dunzeng/more","repo_kind":"official","path":"reward_datasets.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/reward_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"520e1c7389395ede"}},{"code_sha256_prefix":"f43d68f055c6659c","entry":"gpt_winer","repo":"dunzeng/more","repo_kind":"official","path":"evaluation_utils.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f43d68f055c6659c"}},{"code_sha256_prefix":"1b7ba4d146c226aa","entry":"gradient_normalizers","repo":"dunzeng/more","repo_kind":"official","path":"min_norm_solvers.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/min_norm_solvers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b7ba4d146c226aa"}},{"code_sha256_prefix":"53aceea1ad75f96e","entry":"load_dataset","repo":"dunzeng/more","repo_kind":"official","path":"reward_model_inference.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/reward_model_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53aceea1ad75f96e"}},{"code_sha256_prefix":"ec5cde88ee0532ee","entry":"more_data_collator_without_resampling","repo":"dunzeng/more","repo_kind":"official","path":"reward_datasets.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/reward_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec5cde88ee0532ee"}},{"code_sha256_prefix":"5b32b7c64711b73b","entry":"reward_data_collator","repo":"dunzeng/more","repo_kind":"official","path":"reward_datasets.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/reward_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b32b7c64711b73b"}},{"code_sha256_prefix":"bd566700507d4d13","entry":"compute_ppl","repo":"dunzeng/more","repo_kind":"official","path":"evaluation_utils.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd566700507d4d13"}},{"code_sha256_prefix":"4cc0a4e7f078667f","entry":"dialog_to_llama_tokens","repo":"dunzeng/more","repo_kind":"official","path":"llm_inferencing.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/llm_inferencing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cc0a4e7f078667f"}},{"code_sha256_prefix":"9816d50d6a55e67e","entry":"hh_to_dialog","repo":"dunzeng/more","repo_kind":"official","path":"llm_inferencing.py","file_url":"https://github.com/dunzeng/more/blob/HEAD/llm_inferencing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9816d50d6a55e67e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}