{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bayesian-calibration-of-win-rate-estimation","title":"Bayesian Calibration of Win Rate Estimation with LLM Evaluators","arxiv_id":"2411.04424","date":"2024-11-07","proceeding":null,"authors":["Yicheng Gao","Gonghan Xu","Zhe Wang","Arman Cohan"],"abstract":"Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.","url_abs":"https://arxiv.org/abs/2411.04424v1","url_pdf":"https://arxiv.org/pdf/2411.04424v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bayesian-calibration-of-win-rate-estimation","repo_url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"story-generation","task_name":"Story Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.04424","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.04424"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"77526c3df9498aa9","entry":"estimate_bernoulli_parameter","repo":"yale-nlp/bay-calibration-llm-evaluators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77526c3df9498aa9"}},{"code_sha256_prefix":"343ed5023f5900d4","entry":"extract_data_from_file","repo":"yale-nlp/bay-calibration-llm-evaluators","repo_kind":"official","path":"process_logs.py","file_url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators/blob/HEAD/process_logs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"343ed5023f5900d4"}},{"code_sha256_prefix":"09de802166b089fc","entry":"plot_p","repo":"yale-nlp/bay-calibration-llm-evaluators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09de802166b089fc"}},{"code_sha256_prefix":"7fe2951898a41c2d","entry":"sample_q","repo":"yale-nlp/bay-calibration-llm-evaluators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yale-nlp/bay-calibration-llm-evaluators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7fe2951898a41c2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}