{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-language-models-with-language","title":"Training Language Models with Language Feedback at Scale","arxiv_id":"2303.16755","date":"2023-03-28","proceeding":null,"authors":["Jérémy Scheurer","Jon Ander Campos","Tomasz Korbak","Jun Shern Chan","Angelica Chen","Kyunghyun Cho","Ethan Perez"],"abstract":"Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries. Learning from both language and comparison feedback outperforms learning from each alone, achieving human-level summarization performance.","url_abs":"https://arxiv.org/abs/2303.16755v3","url_pdf":"https://arxiv.org/pdf/2303.16755v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-language-models-with-language","repo_url":"https://github.com/jeremyalain/imitation_learning_from_language_feedback","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.16755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jeremyalain/imitation_learning_from_language_feedback","reach":null}],"summary":{"ran_violates":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"61cf51e9037e37a3","entry":"calculate_reward_model_loss","repo":"jeremyalain/imitation_learning_from_language_feedback","repo_kind":"official","path":"reward_model/train_reward_model.py","file_url":"https://github.com/jeremyalain/imitation_learning_from_language_feedback/blob/HEAD/reward_model/train_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61cf51e9037e37a3"}},{"code_sha256_prefix":"b5e40d97f3fdfe96","entry":"evaluate_model_on_validation_set","repo":"jeremyalain/imitation_learning_from_language_feedback","repo_kind":"official","path":"reward_model/train_reward_model.py","file_url":"https://github.com/jeremyalain/imitation_learning_from_language_feedback/blob/HEAD/reward_model/train_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5e40d97f3fdfe96"}},{"code_sha256_prefix":"9545adfafc541c06","entry":"get_predictions","repo":"jeremyalain/imitation_learning_from_language_feedback","repo_kind":"official","path":"reward_model/train_reward_model.py","file_url":"https://github.com/jeremyalain/imitation_learning_from_language_feedback/blob/HEAD/reward_model/train_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9545adfafc541c06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}