{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-13576","title":"Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges","arxiv_id":"2602.13576","date":"2026-02-14","proceeding":null,"authors":["Ruomeng Ding","Yifei Pang","He Sun","Yizhong Wang","Zhiwei Steven Wu","Zhun Deng"],"abstract":"Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-language rubrics and validated on benchmarks. We identify a previously under-recognized vulnerability in this workflow, which we term Rubric-Induced Preference Drift (RIPD). Even when rubric edits pass benchmark validation, they can still produce systematic and directional shifts in a judge's preferences on target domains. Because rubrics serve as a high-level decision interface, such drift can emerge from seemingly natural, criterion-preserving edits and remain difficult to detect through aggregate benchmark metrics or limited spot-checking. We further show this vulnerability can be exploited through rubric-based preference attacks, in which benchmark-compliant rubric edits steer judgments away from a fixed human or trusted reference on target domains, systematically inducing RIPD and reducing target-domain accuracy up to 9.5% (helpfulness) and 27.9% (harmlessness). When these judgments are used to generate preference labels for downstream post-training, the induced bias propagates through alignment pipelines and becomes internalized in trained policies. This leads to persistent and systematic drift in model behavior. Overall, our findings highlight evaluation rubrics as a sensitive and manipulable control interface, revealing a system-level alignment risk that extends beyond evaluator reliability alone. The code is available at: https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface. Warning: Certain sections may contain potentially harmful content that may not be appropriate for all readers.","url_abs":"https://arxiv.org/abs/2602.13576","url_pdf":"https://arxiv.org/pdf/2602.13576","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2602.13576","atlas_url":"https://app.syntology.ai/?focus=2602.13576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.13576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface","reach":null}],"summary":{"unverified":21},"by_repo_kind":{"found_in_text":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f2553208ec910158","entry":"align_two_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/utils_pairwise.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/utils_pairwise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2553208ec910158"}},{"code_sha256_prefix":"e7004c97af9d006f","entry":"collect_error_examples","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_search/search/optimize.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_search/search/optimize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7004c97af9d006f"}},{"code_sha256_prefix":"1db28a5ee819cd01","entry":"compute_stats","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/analyze.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1db28a5ee819cd01"}},{"code_sha256_prefix":"ccb26c1a58a2eb7c","entry":"decide_winner","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/analyze.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccb26c1a58a2eb7c"}},{"code_sha256_prefix":"156a7a83c700039e","entry":"eval_on_test","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_selection/selection.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_selection/selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"156a7a83c700039e"}},{"code_sha256_prefix":"d05a5f62b77b232a","entry":"format_beaver_conversation","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/scores.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d05a5f62b77b232a"}},{"code_sha256_prefix":"1974826b28a222b3","entry":"format_error_examples_for_prompt","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_search/search/optimize.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_search/search/optimize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1974826b28a222b3"}},{"code_sha256_prefix":"40244ed50a2b0259","entry":"get_rubric_id","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_search/search/select_final.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_search/search/select_final.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40244ed50a2b0259"}},{"code_sha256_prefix":"26bdfbc372c5a3a9","entry":"get_token_len","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/generate.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26bdfbc372c5a3a9"}},{"code_sha256_prefix":"9b9ceb2f80a80ac0","entry":"load_gen_file_bon","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/scores.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b9ceb2f80a80ac0"}},{"code_sha256_prefix":"400eeff90e1ba381","entry":"load_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/utils_pairwise.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/utils_pairwise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"400eeff90e1ba381"}},{"code_sha256_prefix":"555478893737fb22","entry":"load_rubrics_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_search/search/select_final.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_search/search/select_final.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"555478893737fb22"}},{"code_sha256_prefix":"f57b8c707f9f56df","entry":"parse_verdict","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/utils_pairwise.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/utils_pairwise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f57b8c707f9f56df"}},{"code_sha256_prefix":"23262afede4a7d29","entry":"pick_best_in_subset","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/select_best.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/select_best.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23262afede4a7d29"}},{"code_sha256_prefix":"d97bc4cdc990894c","entry":"read_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/scores.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d97bc4cdc990894c"}},{"code_sha256_prefix":"ffbf5293f2483abe","entry":"read_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/analyze.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffbf5293f2483abe"}},{"code_sha256_prefix":"f07107c7894990c1","entry":"read_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/generate.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f07107c7894990c1"}},{"code_sha256_prefix":"6a94611fb33432fd","entry":"read_jsonl","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/select_best.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/select_best.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a94611fb33432fd"}},{"code_sha256_prefix":"3e064fc5b4504d36","entry":"sample_instructions_from_dataset","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/generate.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e064fc5b4504d36"}},{"code_sha256_prefix":"6ee262d896a4c9ff","entry":"select_rubric","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"rubrics_selection/selection.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/rubrics_selection/selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ee262d896a4c9ff"}},{"code_sha256_prefix":"de5d416266f706a6","entry":"subset_indices","repo":"ZDCSlab/Rubrics-as-an-Attack-Surface","repo_kind":"found_in_text","path":"downstream_eval/eval/select_best.py","file_url":"https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface/blob/HEAD/downstream_eval/eval/select_best.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de5d416266f706a6"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}