{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-critical-evaluation-of-ai-feedback-for","title":"A Critical Evaluation of AI Feedback for Aligning Large Language Models","arxiv_id":"2402.12366","date":"2024-02-19","proceeding":null,"authors":["Archit Sharma","Sedrick Keh","Eric Mitchell","Chelsea Finn","Kushal Arora","Thomas Kollar"],"abstract":"Reinforcement learning with AI feedback (RLAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step RLAIF pipeline as well as suggestions for making RLAIF maximally useful in practice.","url_abs":"https://arxiv.org/abs/2402.12366v1","url_pdf":"https://arxiv.org/pdf/2402.12366v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-critical-evaluation-of-ai-feedback-for","repo_url":"https://github.com/architsharma97/dpo-rlaif","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"rlaif","method_name":"RLAIF"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sft","method_name":"SFT"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.12366","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12366"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/architsharma97/dpo-rlaif","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16a2450aff4a3577","entry":"dpo_loss","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"trainers.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/trainers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16a2450aff4a3577"}},{"code_sha256_prefix":"12f6803880f9b17e","entry":"extract_anthropic_prompt","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"preference_datasets.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/preference_datasets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12f6803880f9b17e"}},{"code_sha256_prefix":"2a9c061a2b1c35cf","entry":"get_local_dir","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"utils.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a9c061a2b1c35cf"}},{"code_sha256_prefix":"5edbbf974035e91f","entry":"get_local_run_dir","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"utils.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5edbbf974035e91f"}},{"code_sha256_prefix":"3c8dcada6a55875f","entry":"get_logits","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"reward_trainer.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/reward_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8dcada6a55875f"}},{"code_sha256_prefix":"2378ff2496a12f0a","entry":"get_openai_completion","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"ai_completions.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/ai_completions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2378ff2496a12f0a"}},{"code_sha256_prefix":"5530a963e205ad95","entry":"match_instruction_outputs","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"label_ai_preferences.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/label_ai_preferences.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5530a963e205ad95"}},{"code_sha256_prefix":"12be818ac399701d","entry":"train_step","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"reward_trainer.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/reward_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12be818ac399701d"}},{"code_sha256_prefix":"d5e9b37690fe8587","entry":"dump_files","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"generate_samples.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/generate_samples.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5e9b37690fe8587"}},{"code_sha256_prefix":"b1c3a18860528ce9","entry":"eval_loop","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"reward_trainer.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/reward_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1c3a18860528ce9"}},{"code_sha256_prefix":"e7ce4c19888c6df6","entry":"get_remote_file","repo":"architsharma97/dpo-rlaif","repo_kind":"official","path":"utils.py","file_url":"https://github.com/architsharma97/dpo-rlaif/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7ce4c19888c6df6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}