{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ultrafeedback-boosting-language-models-with","title":"UltraFeedback: Boosting Language Models with Scaled AI Feedback","arxiv_id":"2310.01377","date":"2023-10-02","proceeding":null,"authors":["Ganqu Cui","Lifan Yuan","Ning Ding","Guanming Yao","Bingxiang He","Wei Zhu","Yuan Ni","Guotong Xie","Ruobing Xie","Yankai Lin","Zhiyuan Liu","Maosong Sun"],"abstract":"Learning from human feedback has become a pivot technique in aligning large language models (LLMs) with human preferences. However, acquiring vast and premium human feedback is bottlenecked by time, labor, and human capability, resulting in small sizes or limited topics of current datasets. This further hinders feedback learning as well as alignment research within the open-source community. To address this issue, we explore how to go beyond human feedback and collect high-quality \\textit{AI feedback} automatically for a scalable alternative. Specifically, we identify \\textbf{scale and diversity} as the key factors for feedback data to take effect. Accordingly, we first broaden instructions and responses in both amount and breadth to encompass a wider range of user-assistant interactions. Then, we meticulously apply a series of techniques to mitigate annotation biases for more reliable AI feedback. We finally present \\textsc{UltraFeedback}, a large-scale, high-quality, and diversified AI feedback dataset, which contains over 1 million GPT-4 feedback for 250k user-assistant conversations from various aspects. Built upon \\textsc{UltraFeedback}, we align a LLaMA-based model by best-of-$n$ sampling and reinforcement learning, demonstrating its exceptional performance on chat benchmarks. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models, serving as a solid foundation for future feedback learning research. Our data and models are available at https://github.com/thunlp/UltraFeedback.","url_abs":"https://arxiv.org/abs/2310.01377v2","url_pdf":"https://arxiv.org/pdf/2310.01377v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ultrafeedback-boosting-language-models-with","repo_url":"https://github.com/thunlp/ultrafeedback","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"ultrafeedback-boosting-language-models-with","repo_url":"https://github.com/openbmb/ultrafeedback","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"ultrafeedback-boosting-language-models-with","repo_url":"https://github.com/stanfordnlp/pyreft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ultrafeedback-boosting-language-models-with","repo_url":"https://github.com/yale-nlp/comal","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.01377","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.01377"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yale-nlp/comal","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stanfordnlp/pyreft","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openbmb/ultrafeedback","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/ultrafeedback","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d3323a5d2d6ed3e","entry":"calculate_average_rating","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/fix_overall_score_issue.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/fix_overall_score_issue.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d3323a5d2d6ed3e"}},{"code_sha256_prefix":"7aabc2143f0f3e10","entry":"check_score","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/fix_overall_score_issue.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/fix_overall_score_issue.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7aabc2143f0f3e10"}},{"code_sha256_prefix":"a506738814a38c14","entry":"get_eval","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/annotate_critique.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/annotate_critique.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a506738814a38c14"}},{"code_sha256_prefix":"9f8a0e28964db553","entry":"get_eval","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/fix_overall_score_issue.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/fix_overall_score_issue.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f8a0e28964db553"}},{"code_sha256_prefix":"993cf7ccb3239873","entry":"process","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/annotate_preference.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/annotate_preference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"993cf7ccb3239873"}},{"code_sha256_prefix":"2b3cb1aee424f3c8","entry":"annotate","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/annotate_critique.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/annotate_critique.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b3cb1aee424f3c8"}},{"code_sha256_prefix":"38d99fe231928f20","entry":"get_eval","repo":"thunlp/ultrafeedback","repo_kind":"official","path":"src/data_annotation/annotate_preference.py","file_url":"https://github.com/thunlp/ultrafeedback/blob/HEAD/src/data_annotation/annotate_preference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38d99fe231928f20"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}