{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/integrative-decoding-improve-factuality-via","title":"Integrative Decoding: Improve Factuality via Implicit Self-consistency","arxiv_id":"2410.01556","date":"2024-10-02","proceeding":null,"authors":["Yi Cheng","Xiao Liang","Yeyun Gong","Wen Xiao","Song Wang","Yuji Zhang","Wenjun Hou","Kaishuai Xu","Wenge Liu","Wenjie Li","Jian Jiao","Qi Chen","Peng Cheng","Wayne Xiong"],"abstract":"Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sampled response, and then processes them concurrently, with the next token being selected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates self-consistency in the decoding objective. Extensive evaluation shows that ID consistently enhances factuality over a wide range of language models, with substantial improvements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling.","url_abs":"https://arxiv.org/abs/2410.01556v4","url_pdf":"https://arxiv.org/pdf/2410.01556v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"integrative-decoding-improve-factuality-via","repo_url":"https://github.com/yicheng98/integrativedecoding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"truthfulqa","task_name":"TruthfulQA"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.01556","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.01556"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yicheng98/integrativedecoding","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a81cf4eac163a89c","entry":"merge_probs","repo":"yicheng98/integrativedecoding","repo_kind":"official","path":"utils/voting_decoding.py","file_url":"https://github.com/yicheng98/integrativedecoding/blob/HEAD/utils/voting_decoding.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a81cf4eac163a89c"}},{"code_sha256_prefix":"3ecacf48409a6e59","entry":"LlamaForCausalLMWithVD","repo":"yicheng98/integrativedecoding","repo_kind":"official","path":"utils/voting_decoding.py","file_url":"https://github.com/yicheng98/integrativedecoding/blob/HEAD/utils/voting_decoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ecacf48409a6e59"}},{"code_sha256_prefix":"2cb0bc272a6eeae7","entry":"load_sample_responses","repo":"YiCheng98/IntegrativeDecoding","repo_kind":"official","path":"inference_methods.py","file_url":"https://github.com/YiCheng98/IntegrativeDecoding/blob/HEAD/inference_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cb0bc272a6eeae7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}