{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/peer-review-in-llms-automatic-evaluation","title":"PiCO: Peer Review in LLMs based on the Consistency Optimization","arxiv_id":"2402.01830","date":"2024-02-02","proceeding":null,"authors":["Kun-Peng Ning","Shuo Yang","Yu-Yang Liu","Jia-Yu Yao","Zhen-Hui Liu","Yu Wang","Ming Pang","Li Yuan"],"abstract":"Existing large language models (LLMs) evaluation methods typically focus on testing the performance on some closed-environment and domain-specific benchmarks with human annotations. In this paper, we explore a novel unsupervised evaluation direction, utilizing peer-review mechanisms to measure LLMs automatically. In this setting, both open-source and closed-source LLMs lie in the same environment, capable of answering unlabeled questions and evaluating each other, where each LLM's response score is jointly determined by other anonymous ones. To obtain the ability hierarchy among these models, we assign each LLM a learnable capability parameter to adjust the final ranking. We formalize it as a constrained optimization problem, intending to maximize the consistency of each LLM's capabilities and scores. The key assumption behind is that high-level LLM can evaluate others' answers more accurately than low-level ones, while higher-level LLM can also achieve higher response scores. Moreover, we propose three metrics called PEN, CIN, and LIS to evaluate the gap in aligning human rankings. We perform experiments on multiple datasets with these metrics, validating the effectiveness of the proposed approach.","url_abs":"https://arxiv.org/abs/2402.01830v2","url_pdf":"https://arxiv.org/pdf/2402.01830v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"peer-review-in-llms-automatic-evaluation","repo_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"pico","task_name":"PICO"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.01830","atlas_url":"https://app.syntology.ai/?focus=2402.01830","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01830"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs","reach":{"status":"ok"}}],"summary":{"ran":9,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"680d8eb1d4a89560","entry":"calculate_combinations","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/utils.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"680d8eb1d4a89560"}},{"code_sha256_prefix":"1e556153d971820f","entry":"evaluate","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/main_ablation.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/main_ablation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e556153d971820f"}},{"code_sha256_prefix":"9921c5a2fe409afc","entry":"generate_stream_codet5p","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/config/model_codet5p.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/config/model_codet5p.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9921c5a2fe409afc"}},{"code_sha256_prefix":"2d40f5866c066fb9","entry":"get_model_info","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/config/model_registry.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/config/model_registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d40f5866c066fb9"}},{"code_sha256_prefix":"70b8da050907a87c","entry":"pearson_correlation","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/main_ablation.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/main_ablation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70b8da050907a87c"}},{"code_sha256_prefix":"7a9c47a7b5588c1d","entry":"permutation_entropy","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/utils.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a9c47a7b5588c1d"}},{"code_sha256_prefix":"563ed3169d69823e","entry":"permutation_entropy_random","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/utils.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"563ed3169d69823e"}},{"code_sha256_prefix":"dc21db7cf25bfe72","entry":"process_response","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/config/model_chatglm.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/config/model_chatglm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc21db7cf25bfe72"}},{"code_sha256_prefix":"0bf414a2b44babe6","entry":"raise_warning_for_incompatible_cpu_offloading_configuration","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/config/model_adapter.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/config/model_adapter.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bf414a2b44babe6"}},{"code_sha256_prefix":"c35f99e4317a61ad","entry":"transform_and_save_conversation_format","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/assign_judge.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/assign_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c35f99e4317a61ad"}},{"code_sha256_prefix":"7812a2ea4cd6d368","entry":"cal_score","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"con_optimization/main_ablation.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/con_optimization/main_ablation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7812a2ea4cd6d368"}},{"code_sha256_prefix":"f018d75f691e3793","entry":"chat_compeletion_openai","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/gen_model_answer.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/gen_model_answer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f018d75f691e3793"}},{"code_sha256_prefix":"9363dd40f202329a","entry":"load_T5model_and_tokenizer","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/common.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9363dd40f202329a"}},{"code_sha256_prefix":"8feea35ea433b832","entry":"load_model_and_tokenizer","repo":"PKU-YuanGroup/Peer-review-in-LLMs","repo_kind":"official","path":"llm_judge/common.py","file_url":"https://github.com/PKU-YuanGroup/Peer-review-in-LLMs/blob/HEAD/llm_judge/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8feea35ea433b832"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}