{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rl4f-generating-natural-language-feedback","title":"RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs","arxiv_id":"2305.08844","date":"2023-05-15","proceeding":null,"authors":["Afra Feyza Akyürek","Ekin Akyürek","Aman Madaan","Ashwin Kalyan","Peter Clark","Derry Wijaya","Niket Tandon"],"abstract":"Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.","url_abs":"https://arxiv.org/abs/2305.08844v2","url_pdf":"https://arxiv.org/pdf/2305.08844v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rl4f-generating-natural-language-feedback","repo_url":"https://github.com/feyzaakyurek/rl4f","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"},{"task_slug":"text-similarity","task_name":"text similarity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.08844","atlas_url":"https://app.syntology.ai/?focus=2305.08844","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.08844"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/feyzaakyurek/rl4f","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/RL4LMs","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"found_in_text":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"92e1118e4e5f774c","entry":"conjugate_gradient_solver","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92e1118e4e5f774c"}},{"code_sha256_prefix":"2fff9b7850dc9164","entry":"custom_metric_scripting_func","repo":"feyzaakyurek/rl4f","repo_kind":"official","path":"custom_reward.py","file_url":"https://github.com/feyzaakyurek/rl4f/blob/HEAD/custom_reward.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fff9b7850dc9164"}},{"code_sha256_prefix":"91200094734b3131","entry":"find_products","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91200094734b3131"}},{"code_sha256_prefix":"206123801663c69f","entry":"flat_grad","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"206123801663c69f"}},{"code_sha256_prefix":"e38b5971551fdee2","entry":"get_dict_obj","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e38b5971551fdee2"}},{"code_sha256_prefix":"821b0061bc694ddf","entry":"quantile_huber_loss","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"821b0061bc694ddf"}},{"code_sha256_prefix":"b41d462e09610b1c","entry":"to_expand","repo":"allenai/RL4LMs","repo_kind":"found_in_text","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/allenai/RL4LMs/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b41d462e09610b1c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}