{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/does-context-matter-contextualjudgebench-for","title":"Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings","arxiv_id":"2503.15620","date":"2025-03-19","proceeding":null,"authors":["Austin Xu","Srijan Bansal","Yifei Ming","Semih Yavuz","Shafiq Joty"],"abstract":"The large language model (LLM)-as-judge paradigm has been used to meet the demand for a cheap, reliable, and fast evaluation of model outputs during AI system development and post-deployment monitoring. While judge models -- LLMs finetuned to specialize in assessing and critiquing model outputs -- have been touted as general purpose evaluators, they are typically evaluated only on non-contextual scenarios, such as instruction following. The omission of contextual settings -- those where external information is used as context to generate an output -- is surprising given the increasing prevalence of retrieval-augmented generation (RAG) and summarization use cases. Contextual assessment is uniquely challenging, as evaluation often depends on practitioner priorities, leading to conditional evaluation criteria (e.g., comparing responses based on factuality and then considering completeness if they are equally factual). To address the gap, we propose ContextualJudgeBench, a judge benchmark with 2,000 challenging response pairs across eight splits inspired by real-world contextual evaluation scenarios. We build our benchmark with a multi-pronged data construction pipeline that leverages both existing human annotations and model-based perturbations. Our comprehensive study across 11 judge models and 9 general purpose models, reveals that the contextual information and its assessment criteria present a significant challenge to even state-of-the-art models. For example, OpenAI's o1, the best-performing model, barely reaches 55% consistent accuracy.","url_abs":"https://arxiv.org/abs/2503.15620v1","url_pdf":"https://arxiv.org/pdf/2503.15620v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"does-context-matter-contextualjudgebench-for","repo_url":"https://github.com/salesforceairesearch/contextualjudgebench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.15620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.15620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforceairesearch/contextualjudgebench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"e4c43673ff6920c0","entry":"chat_completion_openai","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e4c43673ff6920c0"}},{"code_sha256_prefix":"e04d85200b56e336","entry":"get_ragas_pairwise","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e04d85200b56e336"}},{"code_sha256_prefix":"951279c1a44542a4","entry":"load_prompts_and_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompt_utils.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompt_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"951279c1a44542a4"}},{"code_sha256_prefix":"90d42ba9463e8243","entry":"pairwise_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompts/atla.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompts/atla.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"90d42ba9463e8243"}},{"code_sha256_prefix":"5deb1637623318bb","entry":"pairwise_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompts/glider.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompts/glider.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5deb1637623318bb"}},{"code_sha256_prefix":"92724d6a93f6015f","entry":"pairwise_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompts/offsetbias.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompts/offsetbias.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"92724d6a93f6015f"}},{"code_sha256_prefix":"d6755ed37ba0a463","entry":"pairwise_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompts/prometheus.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompts/prometheus.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d6755ed37ba0a463"}},{"code_sha256_prefix":"38e8813d00261fc4","entry":"pairwise_parsing","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/prompts/selftaught.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/prompts/selftaught.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"38e8813d00261fc4"}},{"code_sha256_prefix":"c89138f7dd4645e9","entry":"read_jsonl","repo":"salesforceairesearch/contextualjudgebench","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/salesforceairesearch/contextualjudgebench/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c89138f7dd4645e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}