{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-we-trust-the-performance-evaluation-of","title":"Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization?","arxiv_id":"2406.17274","date":"2024-06-25","proceeding":null,"authors":["Jianfeng He","Runing Yang","Linlin Yu","Changbin Li","Ruoxi Jia","Feng Chen","Ming Jin","Chang-Tien Lu"],"abstract":"Text summarization, a key natural language generation (NLG) task, is vital in various domains. However, the high cost of inaccurate summaries in risk-critical applications, particularly those involving human-in-the-loop decision-making, raises concerns about the reliability of uncertainty estimation on text summarization (UE-TS) evaluation methods. This concern stems from the dependency of uncertainty model metrics on diverse and potentially conflicting NLG metrics. To address this issue, we introduce a comprehensive UE-TS benchmark incorporating 31 NLG metrics across four dimensions. The benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. We also assess the performance of 14 common uncertainty estimation methods within this benchmark. Our findings emphasize the importance of considering multiple uncorrelated NLG metrics and diverse uncertainty estimation methods to ensure reliable and efficient evaluation of UE-TS techniques. Our code and data are available https://github.com/he159ok/Benchmark-of-Uncertainty-Estimation-Methods-in-Text-Summarization.","url_abs":"https://arxiv.org/abs/2406.17274v2","url_pdf":"https://arxiv.org/pdf/2406.17274v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-we-trust-the-performance-evaluation-of","repo_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.17274","atlas_url":"https://app.syntology.ai/?focus=2406.17274","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.17274"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2fb157422905b822","entry":"compute_sim_score","repo":"he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","repo_kind":"official","path":"src/lm_polygraph/estimators/common.py","file_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization/blob/HEAD/src/lm_polygraph/estimators/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb157422905b822"}},{"code_sha256_prefix":"1a1197e95fb55bd4","entry":"normalize","repo":"he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","repo_kind":"official","path":"understanding_prr.py","file_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization/blob/HEAD/understanding_prr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a1197e95fb55bd4"}},{"code_sha256_prefix":"ae44e53b9d26ee03","entry":"parse_model","repo":"he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","repo_kind":"official","path":"src/lm_polygraph/app/parsers.py","file_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization/blob/HEAD/src/lm_polygraph/app/parsers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae44e53b9d26ee03"}},{"code_sha256_prefix":"bc3cd83bba55ea18","entry":"transfer_general_res_dict","repo":"he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","repo_kind":"official","path":"MyFunc.py","file_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization/blob/HEAD/MyFunc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc3cd83bba55ea18"}},{"code_sha256_prefix":"a41787b7df63a622","entry":"understand_ue_metric","repo":"he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization","repo_kind":"official","path":"understanding_prr.py","file_url":"https://github.com/he159ok/benchmark-of-uncertainty-estimation-methods-in-text-summarization/blob/HEAD/understanding_prr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a41787b7df63a622"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}