{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reward-learning-from-multiple-feedback-types","title":"Reward Learning from Multiple Feedback Types","arxiv_id":"2502.21038","date":"2025-02-28","proceeding":null,"authors":["Yannick Metz","András Geiszl","Raphaël Baur","Mennatallah El-Assady"],"abstract":"Learning rewards from preference feedback has become an important tool in the alignment of agentic models. Preference-based feedback, often implemented as a binary comparison between multiple completions, is an established method to acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process. Despite these potential benefits, learning from different feedback types has yet to be explored extensively. In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a broad set of environments. We present a process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types. Based on the simulated feedback, we investigate the use of types of feedback across ten RL environments and compare them to pure preference-based baselines. We show empirically that diverse types of feedback can be utilized and lead to strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.","url_abs":"https://arxiv.org/abs/2502.21038v1","url_pdf":"https://arxiv.org/pdf/2502.21038v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reward-learning-from-multiple-feedback-types","repo_url":"https://github.com/ymetz/multi-type-feedback","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.21038","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.21038"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ymetz/multi-type-feedback","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2560365dd8eed65a","entry":"calculate_mle_loss","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/networks.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2560365dd8eed65a"}},{"code_sha256_prefix":"736376e150697241","entry":"calculate_mse_loss","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/networks.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"736376e150697241"}},{"code_sha256_prefix":"3da7c4dc9a1697b8","entry":"calculate_pairwise_loss","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/networks.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3da7c4dc9a1697b8"}},{"code_sha256_prefix":"9a80c3f2202ca79b","entry":"create_matrix_vectorized","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/dynamic_rlhf.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/dynamic_rlhf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a80c3f2202ca79b"}},{"code_sha256_prefix":"7b163183389a5fe3","entry":"create_segments","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/generate_feedback_videos.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/generate_feedback_videos.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b163183389a5fe3"}},{"code_sha256_prefix":"1c68deb66b393cf8","entry":"create_segments","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/generate_samples.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/generate_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c68deb66b393cf8"}},{"code_sha256_prefix":"9549a5bef1286f9c","entry":"discounted_sum_numpy","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/feedback_dataset.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/feedback_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9549a5bef1286f9c"}},{"code_sha256_prefix":"e7b2b4e96bf46808","entry":"discounted_sum_numpy","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/generate_feedback_videos.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/generate_feedback_videos.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b2b4e96bf46808"}},{"code_sha256_prefix":"f01c1da3901ed173","entry":"discounted_sum_numpy","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/generate_samples.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/generate_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f01c1da3901ed173"}},{"code_sha256_prefix":"09c37752818feefa","entry":"one_hot_vector","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/dynamic_rlhf.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/dynamic_rlhf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09c37752818feefa"}},{"code_sha256_prefix":"2473e2dac6860122","entry":"one_hot_vector","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/generate_samples.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/generate_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2473e2dac6860122"}},{"code_sha256_prefix":"65ee0ec7c9315f91","entry":"truncated_gaussian_vectorized","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/feedback_dataset.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/feedback_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65ee0ec7c9315f91"}},{"code_sha256_prefix":"b2acb8c3336e78ff","entry":"truncated_uniform_vectorized","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/feedback_dataset.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/feedback_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2acb8c3336e78ff"}},{"code_sha256_prefix":"46e92ab6aa91adb3","entry":"vectorized_one_hot_vector","repo":"ymetz/multi-type-feedback","repo_kind":"official","path":"multi_type_feedback/dynamic_rlhf.py","file_url":"https://github.com/ymetz/multi-type-feedback/blob/HEAD/multi_type_feedback/dynamic_rlhf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46e92ab6aa91adb3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}