{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/eliciting-informative-text-evaluations-with","title":"Eliciting Informative Text Evaluations with Large Language Models","arxiv_id":"2405.15077","date":"2024-05-23","proceeding":null,"authors":["Yuxuan Lu","Shengwei Xu","Yichi Zhang","Yuqing Kong","Grant Schoenebeck"],"abstract":"Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar numbers. We aim to broaden these techniques to the larger domain of text-based reports, drawing on the recent developments in large language models. This vastly increases the applicability of peer prediction mechanisms as textual feedback is the norm in a large variety of feedback channels: peer reviews, e-commerce customer reviews, and comments on social media. We introduce two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM). These mechanisms utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report. Theoretically, we show that when the LLM prediction is sufficiently accurate, our mechanisms can incentivize high effort and truth-telling as an (approximate) Bayesian Nash equilibrium. Empirically, we confirm the efficacy of our mechanisms through experiments conducted on two real datasets: the Yelp review dataset and the ICLR OpenReview dataset. We highlight the results that on the ICLR dataset, our mechanisms can differentiate three quality levels -- human-written reviews, GPT-4-generated reviews, and GPT-3.5-generated reviews in terms of expected scores. Additionally, GSPPM penalizes LLM-generated reviews more effectively than GPPM.","url_abs":"https://arxiv.org/abs/2405.15077v4","url_pdf":"https://arxiv.org/pdf/2405.15077v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"eliciting-informative-text-evaluations-with","repo_url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15077","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15077"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"summary":{"ran":8,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"90432c0bdc5cd3dd","entry":"extract_element_text","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"pdf_management/pdfparser.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/pdf_management/pdfparser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"90432c0bdc5cd3dd"}},{"code_sha256_prefix":"5a1a20f1d1feda8f","entry":"fetch_from_cache","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5a1a20f1d1feda8f"}},{"code_sha256_prefix":"769e6af963c596c3","entry":"gen_two_pos","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"experiments/baseline.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/experiments/baseline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"769e6af963c596c3"}},{"code_sha256_prefix":"a98a2c012cac1092","entry":"generate_cache_key","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a98a2c012cac1092"}},{"code_sha256_prefix":"3b43b1aabeb844e6","entry":"get_article_title","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"pdf_management/pdfparser.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/pdf_management/pdfparser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3b43b1aabeb844e6"}},{"code_sha256_prefix":"b68c58dce0510cc0","entry":"get_section_text","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"pdf_management/pdfparser.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/pdf_management/pdfparser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b68c58dce0510cc0"}},{"code_sha256_prefix":"6c4e4df7a400e23a","entry":"parse_review_summary","repo":"yx-lu/eliciting-informative-text-evaluations-with-large-language-models","repo_kind":"official","path":"experiments/direct.py","file_url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models/blob/HEAD/experiments/direct.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6c4e4df7a400e23a"}},{"code_sha256_prefix":"d6b82917310bb5fd","entry":"remove_half_lines","repo":"yx-lu/eliciting-informative-text-evaluations-with-large-language-models","repo_kind":"official","path":"experiments/direct.py","file_url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models/blob/HEAD/experiments/direct.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d6b82917310bb5fd"}},{"code_sha256_prefix":"3baf07c6128dd911","entry":"save_to_cache","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3baf07c6128dd911"}},{"code_sha256_prefix":"6eb67df219b6a1f1","entry":"to_ascii","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"openreview/gen_dataset.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/openreview/gen_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6eb67df219b6a1f1"}},{"code_sha256_prefix":"1112623cc2238462","entry":"get_iclr2020_reviews","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"openreview/crawl.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/openreview/crawl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1112623cc2238462"}},{"code_sha256_prefix":"f4c891c85cf99011","entry":"get_pridiction_score","repo":"yx-lu/eliciting-informative-text-evaluations-with-large-language-models","repo_kind":"official","path":"experiments/direct.py","file_url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models/blob/HEAD/experiments/direct.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f4c891c85cf99011"}},{"code_sha256_prefix":"bf386b23c0697873","entry":"get_review_summary","repo":"yx-lu/eliciting-informative-text-evaluations-with-large-language-models","repo_kind":"official","path":"experiments/direct.py","file_url":"https://github.com/yx-lu/eliciting-informative-text-evaluations-with-large-language-models/blob/HEAD/experiments/direct.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bf386b23c0697873"}},{"code_sha256_prefix":"fc5eb58baae10b8a","entry":"get_score","repo":"yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models","repo_kind":"official","path":"experiments/baseline.py","file_url":"https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models/blob/HEAD/experiments/baseline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fc5eb58baae10b8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}