{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sure-summarizing-retrievals-using-answer","title":"SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs","arxiv_id":"2404.13081","date":"2024-04-17","proceeding":null,"authors":["Jaehyung Kim","Jaehyun Nam","Sangwoo Mo","Jongjin Park","Sang-Woo Lee","Minjoon Seo","Jung-Woo Ha","Jinwoo Shin"],"abstract":"Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.","url_abs":"https://arxiv.org/abs/2404.13081v1","url_pdf":"https://arxiv.org/pdf/2404.13081v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sure-summarizing-retrievals-using-answer","repo_url":"https://github.com/bbuing9/iclr24_sure","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"sure-summarizing-retrievals-using-answer","repo_url":"https://github.com/ruc-nlpir/flashrag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.13081","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.13081"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bbuing9/iclr24_sure","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruc-nlpir/flashrag","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ce918095f4972ee2","entry":"divide_candidates","repo":"bbuing9/iclr24_sure","repo_kind":"official","path":"functions/gen_candidates.py","file_url":"https://github.com/bbuing9/iclr24_sure/blob/HEAD/functions/gen_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce918095f4972ee2"}},{"code_sha256_prefix":"03519e281359d4ff","entry":"get_choices_sampling","repo":"bbuing9/iclr24_sure","repo_kind":"official","path":"functions/gen_candidates.py","file_url":"https://github.com/bbuing9/iclr24_sure/blob/HEAD/functions/gen_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03519e281359d4ff"}},{"code_sha256_prefix":"edc3ca55e9b88308","entry":"handle_except","repo":"bbuing9/iclr24_sure","repo_kind":"official","path":"functions/gen_candidates.py","file_url":"https://github.com/bbuing9/iclr24_sure/blob/HEAD/functions/gen_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"edc3ca55e9b88308"}},{"code_sha256_prefix":"437b1a6b2bad3170","entry":"normalize_answer","repo":"bbuing9/iclr24_sure","repo_kind":"official","path":"functions/gen_candidates.py","file_url":"https://github.com/bbuing9/iclr24_sure/blob/HEAD/functions/gen_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"437b1a6b2bad3170"}},{"code_sha256_prefix":"cb8f6e583e479768","entry":"post_process_candidate","repo":"bbuing9/iclr24_sure","repo_kind":"official","path":"functions/gen_candidates.py","file_url":"https://github.com/bbuing9/iclr24_sure/blob/HEAD/functions/gen_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb8f6e583e479768"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}