{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/odsum-new-benchmarks-for-open-domain-multi","title":"ODSum: New Benchmarks for Open Domain Multi-Document Summarization","arxiv_id":"2309.08960","date":"2023-09-16","proceeding":null,"authors":["Yijie Zhou","Kejian Shi","Wencai Zhang","Yixin Liu","Yilun Zhao","Arman Cohan"],"abstract":"Open-domain Multi-Document Summarization (ODMDS) is a critical tool for condensing vast arrays of documents into coherent, concise summaries. With a more inter-related document set, there does not necessarily exist a correct answer for the retrieval, making it hard to measure the retrieving performance. We propose a rule-based method to process query-based document summarization datasets into ODMDS datasets. Based on this method, we introduce a novel dataset, ODSum, a sophisticated case with its document index interdependent and often interrelated. We tackle ODMDS with the \\textit{retrieve-then-summarize} method, and the performance of a list of retrievers and summarizers is investigated. Through extensive experiments, we identify variances in evaluation metrics and provide insights into their reliability. We also found that LLMs suffer great performance loss from retrieving errors. We further experimented methods to improve the performance as well as investigate their robustness against imperfect retrieval. We will release our data and code at https://github.com/yale-nlp/ODSum.","url_abs":"https://arxiv.org/abs/2309.08960v1","url_pdf":"https://arxiv.org/pdf/2309.08960v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"odsum-new-benchmarks-for-open-domain-multi","repo_url":"https://github.com/yale-nlp/odsum","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"document-summarization","task_name":"Document Summarization"},{"task_slug":"multi-document-summarization","task_name":"Multi-Document Summarization"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.08960","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.08960"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yale-nlp/odsum","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"564228cc0d894ea5","entry":"load_api_key","repo":"yale-nlp/odsum","repo_kind":"official","path":"code/ablation/refine.py","file_url":"https://github.com/yale-nlp/odsum/blob/HEAD/code/ablation/refine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"564228cc0d894ea5"}},{"code_sha256_prefix":"91f91ba2566b65d7","entry":"get_embedding","repo":"yale-nlp/odsum","repo_kind":"official","path":"code/retreival/retrieve.py","file_url":"https://github.com/yale-nlp/odsum/blob/HEAD/code/retreival/retrieve.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91f91ba2566b65d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}