{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-is-fragile-manipulating","title":"Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models","arxiv_id":"2407.11282","date":"2024-07-15","proceeding":null,"authors":["Qingcheng Zeng","Mingyu Jin","Qinkai Yu","Zhenting Wang","Wenyue Hua","ZiHao Zhou","Guangyan Sun","Yanda Meng","Shiqing Ma","Qifan Wang","Felix Juefei-Xu","Kaize Ding","Fan Yang","Ruixiang Tang","Yongfeng Zhang"],"abstract":"Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. One commonly used method to assess the reliability of LLMs' responses is uncertainty estimation, which gauges the likelihood of their answers being correct. While many studies focus on improving the accuracy of uncertainty estimations for LLMs, our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output. Specifically, the proposed backdoor attack method can alter an LLM's output probability distribution, causing the probability distribution to converge towards an attacker-predefined distribution while ensuring that the top-1 prediction remains unchanged. Our experimental results demonstrate that this attack effectively undermines the model's self-evaluation reliability in multiple-choice questions. For instance, we achieved a 100 attack success rate (ASR) across three different triggering strategies in four models. Further, we investigate whether this manipulation generalizes across different prompts and domains. This work highlights a significant threat to the reliability of LLMs and underscores the need for future defenses against such attacks. The code is available at https://github.com/qcznlp/uncertainty_attack.","url_abs":"https://arxiv.org/abs/2407.11282v3","url_pdf":"https://arxiv.org/pdf/2407.11282v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-is-fragile-manipulating","repo_url":"https://github.com/qcznlp/uncertainty_attack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"uncertainty-is-fragile-manipulating","repo_url":"https://github.com/qcznlp/uncertainty","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.11282","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.11282"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qcznlp/uncertainty_attack","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qcznlp/uncertainty","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f0b2d716de8c900f","entry":"collate_fn","repo":"qcznlp/uncertainty_attack","repo_kind":"official","path":"fine_tuning.py","file_url":"https://github.com/qcznlp/uncertainty_attack/blob/HEAD/fine_tuning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0b2d716de8c900f"}},{"code_sha256_prefix":"6b4ff6a70b9692b6","entry":"tokenize_function","repo":"qcznlp/uncertainty_attack","repo_kind":"official","path":"fine_tuning.py","file_url":"https://github.com/qcznlp/uncertainty_attack/blob/HEAD/fine_tuning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b4ff6a70b9692b6"}},{"code_sha256_prefix":"7a3c0ed788f49496","entry":"load_model","repo":"qcznlp/uncertainty","repo_kind":"official","path":"get_probability_distribution.py","file_url":"https://github.com/qcznlp/uncertainty/blob/HEAD/get_probability_distribution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a3c0ed788f49496"}},{"code_sha256_prefix":"70cca1707a698eb2","entry":"process_data","repo":"qcznlp/uncertainty","repo_kind":"official","path":"get_probability_distribution.py","file_url":"https://github.com/qcznlp/uncertainty/blob/HEAD/get_probability_distribution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70cca1707a698eb2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}