{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/group-preference-optimization-few-shot","title":"Group Preference Optimization: Few-Shot Alignment of Large Language Models","arxiv_id":"2310.11523","date":"2023-10-17","proceeding":null,"authors":["Siyan Zhao","John Dang","Aditya Grover"],"abstract":"Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can be expensive to align for each group, requiring prohibitive amounts of group-specific preference data and computation for real-world use cases. We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner. In GPO, we augment the base LLM with an independent transformer module trained to predict the preferences of a group for the LLM generations. For few-shot learning, we parameterize this module as an in-context autoregressive transformer and train it via meta-learning on several groups. We empirically validate the efficacy of GPO through rigorous evaluations using LLMs with varied sizes on three human opinion adaptation tasks. These tasks involve adapting to the preferences of US demographic groups, global countries, and individual users. Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources, outperforming existing strategies such as in-context steering and fine-tuning methods.","url_abs":"https://arxiv.org/abs/2310.11523v2","url_pdf":"https://arxiv.org/pdf/2310.11523v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"group-preference-optimization-few-shot","repo_url":"https://github.com/jamqd/Group-Preference-Optimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.11523","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.11523"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jamqd/Group-Preference-Optimization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"13efed1531b7955b","entry":"get_logger","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"utils/log.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/utils/log.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13efed1531b7955b"}},{"code_sha256_prefix":"0c12a7acfb5eef30","entry":"get_max_wd","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"train_gpo.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/train_gpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c12a7acfb5eef30"}},{"code_sha256_prefix":"1ce54fd986584f07","entry":"running_average_to_dict","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"utils/log.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/utils/log.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ce54fd986584f07"}},{"code_sha256_prefix":"ac1db62261037f58","entry":"set_available_gpus","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"baselines/utils.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/baselines/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac1db62261037f58"}},{"code_sha256_prefix":"95bbf09cea0d296a","entry":"softmax_normalize","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"train_gpo.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/train_gpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95bbf09cea0d296a"}},{"code_sha256_prefix":"8c8ff957ad1eac3d","entry":"build_mlp","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"models/modules.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/models/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c8ff957ad1eac3d"}},{"code_sha256_prefix":"7a2b06d83e018340","entry":"get_log","repo":"jamqd/Group-Preference-Optimization","repo_kind":"official","path":"utils/log.py","file_url":"https://github.com/jamqd/Group-Preference-Optimization/blob/HEAD/utils/log.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a2b06d83e018340"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}