{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/longeval-guidelines-for-human-evaluation-of","title":"LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization","arxiv_id":"2301.13298","date":"2023-01-30","proceeding":null,"authors":["Kalpesh Krishna","Erin Bransom","Bailey Kuehl","Mohit Iyyer","Pradeep Dasigi","Arman Cohan","Kyle Lo"],"abstract":"While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form summaries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surrounding long-form summaries. We find that 73% of these papers do not perform any human evaluation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreement). Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can we minimize annotator workload while maintaining accurate faithfulness scores? and (3) Do humans benefit from automated alignment between summary and source snippets? We deploy LongEval in annotation studies on two long-form summarization datasets in different domains (SQuALITY and PubMed), and we find that switching to a finer granularity of judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a partial annotation of fine-grained units highly correlates with scores from a full annotation workload (0.89 Kendall's tau using 50% judgments). We release our human judgments, annotation templates, and our software as a Python library for future research.","url_abs":"https://arxiv.org/abs/2301.13298v1","url_pdf":"https://arxiv.org/pdf/2301.13298v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"longeval-guidelines-for-human-evaluation-of","repo_url":"https://github.com/martiansideofthemoon/longeval-summarization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"form","task_name":"Form"}],"methods":[{"method_slug":null,"method_name":"Library"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.13298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.13298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/martiansideofthemoon/longeval-summarization","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"11cb365882095fa0","entry":"compute_fine_scores","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/results_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/results_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11cb365882095fa0"}},{"code_sha256_prefix":"ea0144dd053d59f1","entry":"convert_multivers_format","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/linkage/multivers_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/linkage/multivers_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea0144dd053d59f1"}},{"code_sha256_prefix":"c31d481cb06cfe9f","entry":"get_most_common","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c31d481cb06cfe9f"}},{"code_sha256_prefix":"19d5f60018a23dfe","entry":"get_squality_data_stats","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/results_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/results_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19d5f60018a23dfe"}},{"code_sha256_prefix":"e84b47331b08db59","entry":"jsonl_read","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/preprocessing_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/preprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e84b47331b08db59"}},{"code_sha256_prefix":"7810f9d40d608c62","entry":"masked_binary_cross_entropy_with_logits","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/linkage/longchecker/model.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/linkage/longchecker/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7810f9d40d608c62"}},{"code_sha256_prefix":"51ca133d8ef787db","entry":"pickle_read","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/preprocessing_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/preprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51ca133d8ef787db"}},{"code_sha256_prefix":"ad3eecdca7e17576","entry":"split_sents","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/preprocessing_utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/preprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad3eecdca7e17576"}},{"code_sha256_prefix":"18ccd56b847f897e","entry":"token_counter","repo":"martiansideofthemoon/longeval-summarization","repo_kind":"official","path":"longeval/utils.py","file_url":"https://github.com/martiansideofthemoon/longeval-summarization/blob/HEAD/longeval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18ccd56b847f897e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}