{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rate-score-reward-models-with-imperfect","title":"RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals","arxiv_id":"2410.11348","date":"2024-10-15","proceeding":null,"authors":["David Reber","Sean Richardson","Todd Nief","Cristina Garbacea","Victor Veitch"],"abstract":"Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper we develop Rewrite-based Attribute Treatment Estimator (RATE) as an effective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Importantly, RATE measures the causal effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfactuals examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substantial bias in the estimated sensitivity of the reward model to the attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting twice. We establish the validity of the RATE procedure and show empirically that it is an effective estimator.","url_abs":"https://arxiv.org/abs/2410.11348v3","url_pdf":"https://arxiv.org/pdf/2410.11348v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rate-score-reward-models-with-imperfect","repo_url":"https://github.com/toddnief/rate","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[{"method_slug":"counterfactuals","method_name":"Counterfactuals"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.11348","atlas_url":"https://app.syntology.ai/?focus=2410.11348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/toddnief/RATE","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/toddnief/rate","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"23e857cbe5063d81","entry":"add_json_to_dataset","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/create_dataset.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/create_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23e857cbe5063d81"}},{"code_sha256_prefix":"dcd01497f33a59c1","entry":"calculate_rewrite_effect","repo":"toddnief/rate","repo_kind":"official","path":"src/rate/treatment_effects.py","file_url":"https://github.com/toddnief/rate/blob/HEAD/src/rate/treatment_effects.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcd01497f33a59c1"}},{"code_sha256_prefix":"833e6e13743ea918","entry":"calculate_treatment_effects","repo":"toddnief/rate","repo_kind":"official","path":"src/rate/treatment_effects.py","file_url":"https://github.com/toddnief/rate/blob/HEAD/src/rate/treatment_effects.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"833e6e13743ea918"}},{"code_sha256_prefix":"fb459ec3ad95f049","entry":"create_api_batch","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/gpt4_api.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/gpt4_api.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb459ec3ad95f049"}},{"code_sha256_prefix":"eb6e7eb943de1c18","entry":"load_dataset_from_json","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/utils.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb6e7eb943de1c18"}},{"code_sha256_prefix":"17725b319f32e7e0","entry":"load_scoring_utils","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/scoring_template.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/scoring_template.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17725b319f32e7e0"}},{"code_sha256_prefix":"c1ec67a43b08f1d3","entry":"sample_rewrites","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/utils_plotting.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1ec67a43b08f1d3"}},{"code_sha256_prefix":"b798e8428e27d719","entry":"sample_rewrites_tabular","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/utils_plotting.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b798e8428e27d719"}},{"code_sha256_prefix":"4f84c916f8ee3969","entry":"serialize_experiment_template","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/utils.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f84c916f8ee3969"}},{"code_sha256_prefix":"f00425bfab46076f","entry":"treatment_effects_pipeline","repo":"toddnief/rate","repo_kind":"official","path":"src/rate/treatment_effects.py","file_url":"https://github.com/toddnief/rate/blob/HEAD/src/rate/treatment_effects.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f00425bfab46076f"}},{"code_sha256_prefix":"5c47a0d5dbae9173","entry":"synthetic_plot","repo":"toddnief/RATE","repo_kind":"official","path":"src/rate/utils_plotting.py","file_url":"https://github.com/toddnief/RATE/blob/HEAD/src/rate/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c47a0d5dbae9173"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}