{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-23461","title":"Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance","arxiv_id":"2512.23461","date":"2025-12-29","proceeding":"ICLR","authors":["Zhuo Li","Pengyu Cheng","Zhechao Yu","Feifei Tong","Anningzhe Gao","Tsung-Hui Chang","Xiang Wan","Erchao Zhao","Xiaoxi Jiang","Guanjun Jiang"],"abstract":"Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models (LLMs) with human values. However, RM training data is commonly recognized as low-quality, containing inductive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human-preferred but with more words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, \\textit{e.g.}, Pearson coefficients. To mitigate more complex and diverse inductive biases in reward modeling, we introduce a novel information-theoretic debiasing method called \\textbf{D}ebiasing via \\textbf{I}nformation optimization for \\textbf{R}M (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI between RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effectiveness of DIR with three types of inductive biases: \\textit{response length}, \\textit{sycophancy}, and \\textit{format}. We discover that DIR not only effectively mitigates target inductive biases but also enhances RLHF performance across diverse benchmarks, yielding better generalization abilities. The code and training recipes are available at https://github.com/Qwen-Applications/DIR.","url_abs":"https://arxiv.org/abs/2512.23461","url_pdf":"https://arxiv.org/pdf/2512.23461","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2512.23461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.23461"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Qwen-Applications/DIR","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":17},"by_repo_kind":{"found_in_text":{"samples":17,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"47ee1e29f8388bb2","entry":"build_dataset","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/load_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/load_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47ee1e29f8388bb2"}},{"code_sha256_prefix":"d5c8de0d3609ddc2","entry":"build_dataset_SK","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/load_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/load_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5c8de0d3609ddc2"}},{"code_sha256_prefix":"c7dd18f13397a20b","entry":"build_dataset_UF","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/load_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/load_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7dd18f13397a20b"}},{"code_sha256_prefix":"7b30b37de5cc56e7","entry":"build_eval_dataset_biasbench","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/eval_biasbench.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/eval_biasbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b30b37de5cc56e7"}},{"code_sha256_prefix":"629048a34404782d","entry":"build_model","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/eval_biasbench.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/eval_biasbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"629048a34404782d"}},{"code_sha256_prefix":"33044ac6954b842b","entry":"build_ood_eval_dataset","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/load_eval_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/load_eval_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33044ac6954b842b"}},{"code_sha256_prefix":"4075415e3c419aef","entry":"build_unified_eval_dataset","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/load_eval_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/load_eval_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4075415e3c419aef"}},{"code_sha256_prefix":"4014c6ad9f997c4a","entry":"calculate_accuracy_by_domain","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/eval_biasbench.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/eval_biasbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4014c6ad9f997c4a"}},{"code_sha256_prefix":"7710f1274273fc6c","entry":"compute_metrics","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/utils.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7710f1274273fc6c"}},{"code_sha256_prefix":"aed395f45af87c9c","entry":"get_trainable_weights","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/utils.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aed395f45af87c9c"}},{"code_sha256_prefix":"f8e7d855924e7e64","entry":"is_lora_model","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/utils.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8e7d855924e7e64"}},{"code_sha256_prefix":"6d8002bad0675e8b","entry":"load_eval_dataset","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/load_eval_datasets.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/load_eval_datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8002bad0675e8b"}},{"code_sha256_prefix":"acb09d50429275fd","entry":"load_responses","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/arean_hard/arean_evaluation.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/arean_hard/arean_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acb09d50429275fd"}},{"code_sha256_prefix":"be08a58ad66166a5","entry":"parse_judgment","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"rm_eval/arean_hard/arean_evaluation.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/rm_eval/arean_hard/arean_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be08a58ad66166a5"}},{"code_sha256_prefix":"17922d5baec5883a","entry":"post_process_HS3","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/debias_trainer_sycophancy.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/debias_trainer_sycophancy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17922d5baec5883a"}},{"code_sha256_prefix":"97f43d7c928e992e","entry":"post_process_RBv1","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/debias_trainer_length.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/debias_trainer_length.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97f43d7c928e992e"}},{"code_sha256_prefix":"66d572c3fb9e65bc","entry":"post_process_RMBench","repo":"Qwen-Applications/DIR","repo_kind":"found_in_text","path":"reward_models/debias_trainer_length.py","file_url":"https://github.com/Qwen-Applications/DIR/blob/HEAD/reward_models/debias_trainer_length.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66d572c3fb9e65bc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}