{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-20461","title":"Document Summarization with Conformal Importance Guarantees","arxiv_id":"2509.20461","date":"2025-09-24","proceeding":"NeurIPS","authors":["Bruce Kuwahara","Chen-Yuan Lin","Xiao Shi Huang","Kin Kwan Leung","Jullian Arta Yapeter","Ilya Stanevich","Felipe Perez","Jesse C. Cresswell"],"abstract":"Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at https://github.com/layer6ai-labs/conformal-importance-summarization.","url_abs":"https://arxiv.org/abs/2509.20461","url_pdf":"https://arxiv.org/pdf/2509.20461","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.20461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.20461"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/layer6ai-labs/conformal-importance-summarization","reach":null}],"summary":{"ran_honours":1,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"63b0e2f168975d3f","entry":"sentence_position","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63b0e2f168975d3f"}},{"code_sha256_prefix":"e438363cbf7ad338","entry":"ConfidenceMethod","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e438363cbf7ad338"}},{"code_sha256_prefix":"cb9da16022470140","entry":"GUSUM","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb9da16022470140"}},{"code_sha256_prefix":"f7b2882edc738327","entry":"MethodFactory","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7b2882edc738327"}},{"code_sha256_prefix":"55525136b420b7af","entry":"numerical_token_ratio","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55525136b420b7af"}},{"code_sha256_prefix":"59df380ab0d14021","entry":"proper_noun_ratio","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59df380ab0d14021"}},{"code_sha256_prefix":"bf7d3fe77b4b9878","entry":"sentence_length","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf7d3fe77b4b9878"}},{"code_sha256_prefix":"ea35a53d7557a01a","entry":"sentence_ranking","repo":"layer6ai-labs/conformal-importance-summarization","repo_kind":"found_in_text","path":"src/scoring/gusum.py","file_url":"https://github.com/layer6ai-labs/conformal-importance-summarization/blob/HEAD/src/scoring/gusum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea35a53d7557a01a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}