{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prd-peer-rank-and-discussion-improve-large","title":"PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations","arxiv_id":"2307.02762","date":"2023-07-06","proceeding":null,"authors":["Ruosen Li","Teerth Patel","Xinya Du"],"abstract":"Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering. More specifically, they use the recognized \"strongest\" LLM as the evaluator, which conducts pairwise comparisons of candidate models' answers and provides a ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the educational domain (Cho & MacArthur, 2011; Walsh, 2014) to improve LLM-based evaluations. Specifically, we propose (1) the peer rank (PR) algorithm that takes into account each peer LLM's pairwise preferences of all answer pairs, and outputs a final ranking of models; and (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on the preferences of two answers. We conduct experiments on two benchmark datasets. We find that our approaches achieve higher accuracy and align better with human judgments. Interestingly, PR can induce a relatively accurate self-ranking of models under the anonymous setting, where each model's name is unrevealed. Our work provides space to explore evaluating models that are hard to compare for humans.","url_abs":"https://arxiv.org/abs/2307.02762v3","url_pdf":"https://arxiv.org/pdf/2307.02762v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prd-peer-rank-and-discussion-improve-large","repo_url":"https://github.com/bcdnlp/prd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"open-question","task_name":"Open-Ended Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.02762","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.02762"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bcdnlp/prd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_violates":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f296a758b3b2f6d2","entry":"get_eval","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_rank/eval_bard_review.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_rank/eval_bard_review.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f296a758b3b2f6d2"}},{"code_sha256_prefix":"e5264b01e60f3d3d","entry":"get_json_list","repo":"bcdnlp/prd","repo_kind":"official","path":"human_annotation/generate_mturk_input.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/human_annotation/generate_mturk_input.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5264b01e60f3d3d"}},{"code_sha256_prefix":"b012cf9e1a1a190c","entry":"get_review_jsons","repo":"bcdnlp/prd","repo_kind":"official","path":"human_annotation/generate_mturk_input.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/human_annotation/generate_mturk_input.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b012cf9e1a1a190c"}},{"code_sha256_prefix":"dfd1c0fa50d644f7","entry":"history_formatting","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/utils.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfd1c0fa50d644f7"}},{"code_sha256_prefix":"8049b382893c73dd","entry":"parse_score","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_rank/eval_bard_review.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_rank/eval_bard_review.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8049b382893c73dd"}},{"code_sha256_prefix":"2155c3b1b19d8ab9","entry":"parse_three_class_score","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_rank/eval_bard_review.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_rank/eval_bard_review.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2155c3b1b19d8ab9"}},{"code_sha256_prefix":"6561343e8ca358e0","entry":"prompt_formatting","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/openai_api.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/openai_api.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6561343e8ca358e0"}},{"code_sha256_prefix":"6eb61a17cbd928fc","entry":"result_extraction","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/reviews_lfqa.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/reviews_lfqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6eb61a17cbd928fc"}},{"code_sha256_prefix":"acbd88de103b84d2","entry":"result_extraction","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/utils.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acbd88de103b84d2"}},{"code_sha256_prefix":"249a13545ead89d5","entry":"add_prompt_each_turn_lfqa","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/prompt_generation.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/prompt_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"249a13545ead89d5"}},{"code_sha256_prefix":"306d213aec7cb87f","entry":"call","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/openai_api.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/openai_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"306d213aec7cb87f"}},{"code_sha256_prefix":"6e921e9df603b55a","entry":"prompt_discuss_generation_lfqa","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/prompt_generation.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/prompt_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e921e9df603b55a"}},{"code_sha256_prefix":"73ecb69ec08eb250","entry":"prompt_review_generation_lfqa","repo":"bcdnlp/prd","repo_kind":"official","path":"peer_discussion/prompt_generation.py","file_url":"https://github.com/bcdnlp/prd/blob/HEAD/peer_discussion/prompt_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73ecb69ec08eb250"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}