{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-model-preference-evaluation-with","title":"Language Model Preference Evaluation with Multiple Weak Evaluators","arxiv_id":"2410.12869","date":"2024-10-14","proceeding":null,"authors":["Zhengyu Hu","Jieyu Zhang","Zhihan Xiong","Alexander Ratner","Hui Xiong","Ranjay Krishna"],"abstract":"Despite the remarkable success of Large Language Models (LLMs), evaluating their outputs' quality regarding *preference* remains a critical challenge. Existing works usually leverage an LLM as the judge for comparing LLMs' output pairwisely, yet such model-based evaluator is *weak evaluator* due to *conflicting preference*, i.e., output A is better than B, B than C, but C than A, causing contradictory evaluation results. To address this, we introduce GED (Preference Graph Ensemble and Denoise), a novel approach that leverages multiple model-based evaluators to construct preference graphs, and then ensemble and denoise these graphs for better, non-contradictory evaluation results. In particular, our method consists of two primary stages: aggregating evaluations into a unified graph and applying a denoising process to eliminate cyclic inconsistencies, ensuring a directed acyclic graph (DAG) structure. We provide theoretical guarantees for our framework, demonstrating its efficacy in recovering the ground truth preference structure. Extensive experiments on ten benchmarks demonstrate GED's superiority in three applications: model ranking, response selection, and model alignment tasks. Notably, GED combines small LLM evaluators (e.g., Llama3-8B, Mistral-7B, Qwen2-7B) to outperform stronger ones (e.g., Qwen2-72B), showcasing its effectiveness in enhancing evaluation reliability and improving model performance.","url_abs":"https://arxiv.org/abs/2410.12869v3","url_pdf":"https://arxiv.org/pdf/2410.12869v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-model-preference-evaluation-with","repo_url":"https://github.com/ppsmk388/GED","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.12869","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12869"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ppsmk388/GED","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"2eb98bc901715d22","entry":"generate_prompt","repo":"ppsmk388/GED","repo_kind":"official","path":"answer_gen/vllm_generate.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/answer_gen/vllm_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2eb98bc901715d22"}},{"code_sha256_prefix":"ba7818748c5a71fa","entry":"load_json","repo":"ppsmk388/GED","repo_kind":"official","path":"answer_gen/vllm_generate.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/answer_gen/vllm_generate.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba7818748c5a71fa"}},{"code_sha256_prefix":"dfda15dec896b6a7","entry":"perm2ranking","repo":"ppsmk388/GED","repo_kind":"official","path":"rank_utility/ranking_utils.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/rank_utility/ranking_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfda15dec896b6a7"}},{"code_sha256_prefix":"0d1820684bd88236","entry":"rank_ensemble_raw_graph_to_rank","repo":"ppsmk388/GED","repo_kind":"official","path":"tool.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d1820684bd88236"}},{"code_sha256_prefix":"1650a7db63685a24","entry":"ranking_to_score","repo":"ppsmk388/GED","repo_kind":"official","path":"rank_utility/ranking_utils.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/rank_utility/ranking_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1650a7db63685a24"}},{"code_sha256_prefix":"f0a220c0d6d7a42f","entry":"update_rank_list","repo":"ppsmk388/GED","repo_kind":"official","path":"tool.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0a220c0d6d7a42f"}},{"code_sha256_prefix":"85fd19247e8f1c35","entry":"decode_get_item","repo":"ppsmk388/GED","repo_kind":"official","path":"answer_gen/vllm_generate.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/answer_gen/vllm_generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85fd19247e8f1c35"}},{"code_sha256_prefix":"136a7f62c9c037c7","entry":"get_mask","repo":"ppsmk388/GED","repo_kind":"official","path":"rank_utility/synth_ranking_utils.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/rank_utility/synth_ranking_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"136a7f62c9c037c7"}},{"code_sha256_prefix":"1e03004622b64f89","entry":"list_update","repo":"ppsmk388/GED","repo_kind":"official","path":"rank_gen_model_ranking.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/rank_gen_model_ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e03004622b64f89"}},{"code_sha256_prefix":"b833eb110830e983","entry":"list_update","repo":"ppsmk388/GED","repo_kind":"official","path":"rank_gen_response.py","file_url":"https://github.com/ppsmk388/GED/blob/HEAD/rank_gen_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b833eb110830e983"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}