{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semantic-density-uncertainty-quantification","title":"Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space","arxiv_id":"2405.13845","date":"2024-05-22","proceeding":null,"authors":["Xin Qiu","Risto Miikkulainen"],"abstract":"With the widespread application of Large Language Models (LLMs) to various domains, concerns regarding the trustworthiness of LLMs in safety-critical scenarios have been raised, due to their unpredictable tendency to hallucinate and generate misinformation. Existing LLMs do not have an inherent functionality to provide the users with an uncertainty/confidence metric for each response it generates, making it difficult to evaluate trustworthiness. Although several studies aim to develop uncertainty quantification methods for LLMs, they have fundamental limitations, such as being restricted to classification tasks, requiring additional training and data, considering only lexical instead of semantic information, and being prompt-wise but not response-wise. A new framework is proposed in this paper to address these issues. Semantic density extracts uncertainty/confidence information for each response from a probability distribution perspective in semantic space. It has no restriction on task types and is \"off-the-shelf\" for new models and tasks. Experiments on seven state-of-the-art LLMs, including the latest Llama 3 and Mixtral-8x22B models, on four free-form question-answering benchmarks demonstrate the superior performance and robustness of semantic density compared to prior approaches.","url_abs":"https://arxiv.org/abs/2405.13845v3","url_pdf":"https://arxiv.org/pdf/2405.13845v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semantic-density-uncertainty-quantification","repo_url":"https://github.com/cognizant-ai-labs/semantic-density-paper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"misinformation","task_name":"Misinformation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13845","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13845"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cognizant-ai-labs/semantic-density-paper","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lorenzkuhn/semantic","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran_fixture":2,"ran":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"huggingface_replacement/modeling_llama2.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/huggingface_replacement/modeling_llama2.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"huggingface_replacement/modeling_llama2.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/huggingface_replacement/modeling_llama2.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"382e62efddba975b","entry":"clean_text","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"experiment_code/get_semantic_density_full_beam_search_unique_datasets_temperature.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/experiment_code/get_semantic_density_full_beam_search_unique_datasets_temperature.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"382e62efddba975b"}},{"code_sha256_prefix":"707426419c228afb","entry":"get_log_likelihood_variance","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"experiment_code/compute_confidence_measure_beam_search_unique_temperature.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/experiment_code/compute_confidence_measure_beam_search_unique_temperature.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"707426419c228afb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"huggingface_replacement/modeling_llama2.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/huggingface_replacement/modeling_llama2.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"d9308d9bf18cd072","entry":"load_balancing_loss_func","repo":"cognizant-ai-labs/semantic-density-paper","repo_kind":"official","path":"huggingface_replacement/modeling_mixtral.py","file_url":"https://github.com/cognizant-ai-labs/semantic-density-paper/blob/HEAD/huggingface_replacement/modeling_mixtral.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9308d9bf18cd072"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}