{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measurement-integrity-in-peer-prediction-a","title":"Measurement Integrity in Peer Prediction: A Peer Assessment Case Study","arxiv_id":"2108.05521","date":"2021-08-12","proceeding":null,"authors":["Noah Burrell","Grant Schoenebeck"],"abstract":"We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many natural applications. Like robustness against strategic reporting, the property that has been the primary focus of the peer prediction literature, measurement integrity is an important consideration for understanding the practical performance of peer prediction mechanisms. We perform computational experiments, both with an agent-based model and with real data, to empirically evaluate peer prediction mechanisms according to both of these important properties. Our evaluations simulate the application of peer prediction mechanisms to peer assessment -- a setting in which ex post fairness concerns are particularly salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate significant measurement integrity in our experiments. We also find that theoretical properties concerning robustness against strategic reporting are somewhat noisy predictors of empirical performance. Further, there is an apparent trade-off between our two dimensions of analysis. The best-performing mechanisms in terms of measurement integrity are highly susceptible to strategic reporting. Ultimately, however, we show that supplementing mechanisms with realistic parametric statistical models can, in some cases, improve performance along both dimensions of our analysis and result in mechanisms that strike the best balance between them.","url_abs":"https://arxiv.org/abs/2108.05521v2","url_pdf":"https://arxiv.org/pdf/2108.05521v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"measurement-integrity-in-peer-prediction-a","repo_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2108.05521","atlas_url":"https://app.syntology.ai/?focus=2108.05521","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.05521"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d1909ce02ff87f4","entry":"assign_graders","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/grading.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/grading.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d1909ce02ff87f4"}},{"code_sha256_prefix":"f6db8a0c9b2df40d","entry":"assign_graders_dmi_clusters","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/grading_dmi.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/grading_dmi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6db8a0c9b2df40d"}},{"code_sha256_prefix":"42530c6f944a9ea7","entry":"aucs_mse","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/evaluation.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42530c6f944a9ea7"}},{"code_sha256_prefix":"c6ee91443da08370","entry":"get_grading_dict","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/grading.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/grading.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ee91443da08370"}},{"code_sha256_prefix":"7300d38f768b29f9","entry":"pts_mechanism","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/mechanisms/peer_truth_serum.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/mechanisms/peer_truth_serum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7300d38f768b29f9"}},{"code_sha256_prefix":"5d37e0c8c4083c02","entry":"roc_auc","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/evaluation.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d37e0c8c4083c02"}},{"code_sha256_prefix":"42cafbeec2348b4a","entry":"roc_auc_strategic","repo":"burrelln/Measurement-Integrity-and-Peer-Assessment","repo_kind":"official","path":"model_code/evaluation.py","file_url":"https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment/blob/HEAD/model_code/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42cafbeec2348b4a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}