{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shifting-attention-to-relevance-towards-the","title":"Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models","arxiv_id":"2307.01379","date":"2023-07-03","proceeding":null,"authors":["Jinhao Duan","Hao Cheng","Shiqi Wang","Alex Zavalny","Chenan Wang","Renjing Xu","Bhavya Kailkhura","Kaidi Xu"],"abstract":"Large Language Models (LLMs) show promising results in language generation and instruction following but frequently \"hallucinate\", making their outputs less reliable. Despite Uncertainty Quantification's (UQ) potential solutions, implementing it accurately within LLMs is challenging. Our research introduces a simple heuristic: not all tokens in auto-regressive LLM text equally represent the underlying meaning, as \"linguistic redundancy\" often allows a few keywords to convey the essence of long sentences. However, current methods underestimate this inequality when assessing uncertainty, causing tokens with limited semantics to be equally or excessively weighted in UQ. To correct this, we propose Shifting Attention to more Relevant (SAR) components at both token- and sentence-levels for better UQ. We conduct extensive experiments involving a range of popular \"off-the-shelf\" LLMs, such as Vicuna, WizardLM, and LLaMA-2-chat, with model sizes extending up to 33B parameters. We evaluate various free-form question-answering tasks, encompassing domains such as reading comprehension, science Q&A, and medical Q&A. Our experimental results, coupled with a comprehensive demographic analysis, demonstrate the superior performance of SAR. The code is available at https://github.com/jinhaoduan/SAR.","url_abs":"https://arxiv.org/abs/2307.01379v3","url_pdf":"https://arxiv.org/pdf/2307.01379v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shifting-attention-to-relevance-towards-the","repo_url":"https://github.com/jinhaoduan/sar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"shifting-attention-to-relevance-towards-the","repo_url":"https://github.com/jinhaoduan/shifting-attention-to-relevance","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"shifting-attention-to-relevance-towards-the","repo_url":"https://github.com/iinemo/lm-polygraph","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"opt","method_name":"OPT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.01379","atlas_url":"https://app.syntology.ai/?focus=2307.01379","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.01379"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iinemo/lm-polygraph","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinhaoduan/sar","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinhaoduan/shifting-attention-to-relevance","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"baf52bd8c74211f8","entry":"sar","repo":"jinhaoduan/sar","repo_kind":"official","path":"src/compute_uncertainty.py","file_url":"https://github.com/jinhaoduan/sar/blob/HEAD/src/compute_uncertainty.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"baf52bd8c74211f8"}},{"code_sha256_prefix":"9ec4f3a16dcdb560","entry":"sentence_sar","repo":"jinhaoduan/sar","repo_kind":"official","path":"src/compute_uncertainty.py","file_url":"https://github.com/jinhaoduan/sar/blob/HEAD/src/compute_uncertainty.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ec4f3a16dcdb560"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}