{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-07936","title":"Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation","arxiv_id":"2606.07936","date":"2026-06-06","proceeding":null,"authors":["Katelyn Xiaoying Mei","Yi-Li Hsu","Minjoon Choi","Zongwan Cao","Chenjun Xu","Bingbing Wen","Su Lin Blodgett","Lucy Lu Wang"],"abstract":"Human evaluation plays a critical role in assessing the quality of generated text. However, the reliability and reproducibility of these evaluations depend on transparent and well-documented protocols -- details that are frequently missing in current practice. In this work, we conduct a large-scale analysis of human evaluation protocols for evaluating long-form generation tasks in *CL conference publications from 2023--2025, including a full manual review of 284 papers and LLM-assisted analysis for another 1.8k+ papers. We define a set of 20 reportable criteria related to reproducibility of human evaluation studies, and apply these criteria to systematically examine reporting norms and practices within the community. We find widespread under-reporting of important aspects of human evaluation study design, leading to ambiguity about what was measured and how, who contributed judgments, and how judgments should be interpreted. Based on these findings, we outline actionable recommendations to support more transparent and reproducible reporting in future research. Our analysis code and annotated dataset can be found at: https://github.com/larchlab/Illusions-of-the-Gold-Standard","url_abs":"https://arxiv.org/abs/2606.07936","url_pdf":"https://arxiv.org/pdf/2606.07936","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.07936"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard","reach":null}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6f867f364d45edd5","entry":"create_prompt","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"llm_label_v5.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/llm_label_v5.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f867f364d45edd5"}},{"code_sha256_prefix":"512939b4144f4510","entry":"generate_and_parse","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"llm_label_v5.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/llm_label_v5.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"512939b4144f4510"}},{"code_sha256_prefix":"4ab7af3cc40d03e7","entry":"map_paper_id_to_path","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"llm_annotation_chunks.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/llm_annotation_chunks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ab7af3cc40d03e7"}},{"code_sha256_prefix":"8af5343063129578","entry":"evaluate_all_combined","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"evaluate_agreement.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/evaluate_agreement.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8af5343063129578"}},{"code_sha256_prefix":"58565dcde2ee73e9","entry":"evaluate_binary_pair","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"evaluate_agreement.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/evaluate_agreement.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58565dcde2ee73e9"}},{"code_sha256_prefix":"d2c524b38ae1bb03","entry":"evaluate_multi_pair","repo":"larchlab/Illusions-of-the-Gold-Standard","repo_kind":"found_in_text","path":"evaluate_agreement.py","file_url":"https://github.com/larchlab/Illusions-of-the-Gold-Standard/blob/HEAD/evaluate_agreement.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2c524b38ae1bb03"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}