{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-bayesian-active-learning-for-preference","title":"Deep Bayesian Active Learning for Preference Modeling in Large Language Models","arxiv_id":"2406.10023","date":"2024-06-14","proceeding":null,"authors":["Luckeciano C. Melo","Panagiotis Tigas","Alessandro Abate","Yarin Gal"],"abstract":"Leveraging human preferences for steering the behavior of Large Language Models (LLMs) has demonstrated notable success in recent years. Nonetheless, data selection and labeling are still a bottleneck for these systems, particularly at large scale. Hence, selecting the most informative points for acquiring human feedback may considerably reduce the cost of preference labeling and unleash the further development of LLMs. Bayesian Active Learning provides a principled framework for addressing this challenge and has demonstrated remarkable success in diverse settings. However, previous attempts to employ it for Preference Modeling did not meet such expectations. In this work, we identify that naive epistemic uncertainty estimation leads to the acquisition of redundant samples. We address this by proposing the Bayesian Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space spanned by the employed LLM. Notably, our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference labels in two popular human preference datasets and exceeds previous stochastic Bayesian acquisition policies.","url_abs":"https://arxiv.org/abs/2406.10023v2","url_pdf":"https://arxiv.org/pdf/2406.10023v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-bayesian-active-learning-for-preference","repo_url":"https://github.com/luckeciano/bal-pm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.10023","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10023"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/luckeciano/BAL-PM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luckeciano/bal-pm","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"70c52a81fae4c360","entry":"compute_entropy_stats","repo":"luckeciano/BAL-PM","repo_kind":"official","path":"uqlrm/generate_sentence_entropies.py","file_url":"https://github.com/luckeciano/BAL-PM/blob/HEAD/uqlrm/generate_sentence_entropies.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70c52a81fae4c360"}},{"code_sha256_prefix":"cbb32079ed8b32f4","entry":"get_activation_function","repo":"luckeciano/BAL-PM","repo_kind":"official","path":"uqlrm/modules/mc_dropout_reward_mlp.py","file_url":"https://github.com/luckeciano/BAL-PM/blob/HEAD/uqlrm/modules/mc_dropout_reward_mlp.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbb32079ed8b32f4"}},{"code_sha256_prefix":"ffaae593b5909928","entry":"get_entropies","repo":"luckeciano/BAL-PM","repo_kind":"official","path":"uqlrm/generate_sentence_entropies.py","file_url":"https://github.com/luckeciano/BAL-PM/blob/HEAD/uqlrm/generate_sentence_entropies.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffaae593b5909928"}},{"code_sha256_prefix":"127f543d1364a73b","entry":"DropoutLinear_forward","repo":"luckeciano/BAL-PM","repo_kind":"official","path":"uqlrm/modules/mc_dropout_reward_mlp.py","file_url":"https://github.com/luckeciano/BAL-PM/blob/HEAD/uqlrm/modules/mc_dropout_reward_mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"127f543d1364a73b"}},{"code_sha256_prefix":"7a7d51a3dd2dbbc7","entry":"create_datasets","repo":"luckeciano/BAL-PM","repo_kind":"official","path":"uqlrm/adapter_ensemble_reward_training.py","file_url":"https://github.com/luckeciano/BAL-PM/blob/HEAD/uqlrm/adapter_ensemble_reward_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a7d51a3dd2dbbc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}