{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-estimation-and-quantification-for","title":"Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach","arxiv_id":"2404.15993","date":"2024-04-24","proceeding":null,"authors":["Linyu Liu","Yu Pan","Xiaocheng Li","Guanting Chen"],"abstract":"In this paper, we study the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem, a relevant yet underexplored area in existing literature. We then propose a supervised approach that leverages labeled datasets to estimate the uncertainty in LLMs' responses. Based on the formulation, we illustrate the difference between the uncertainty estimation for LLMs and that for standard ML models and explain why the hidden neurons of the LLMs may contain uncertainty information. Our designed approach demonstrates the benefits of utilizing hidden activations to enhance uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. We distinguish the uncertainty estimation task from the uncertainty calibration task and show that better uncertainty estimation leads to better calibration performance. Furthermore, our method is easy to implement and adaptable to different levels of model accessibility including black box, grey box, and white box.","url_abs":"https://arxiv.org/abs/2404.15993v4","url_pdf":"https://arxiv.org/pdf/2404.15993v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-estimation-and-quantification-for","repo_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.15993","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.15993"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":5},"by_repo_kind":{"official":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"b90f1534d7fd6f84","entry":"calculate_ece","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/calibration_metrics.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/calibration_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b90f1534d7fd6f84"}},{"code_sha256_prefix":"ee3bfb38567a7330","entry":"calculate_nll","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/calibration_metrics.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/calibration_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee3bfb38567a7330"}},{"code_sha256_prefix":"96f7a0b4bbfc3427","entry":"calculate_nll_ece","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/calibration_metrics.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/calibration_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96f7a0b4bbfc3427"}},{"code_sha256_prefix":"9d13bc6d9727abfd","entry":"coqa_formatter","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/data_entry.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/data_entry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d13bc6d9727abfd"}},{"code_sha256_prefix":"bd39f0d24eaa33a4","entry":"correlation_select_k_features","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_feature_selection.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_feature_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd39f0d24eaa33a4"}},{"code_sha256_prefix":"50864f75b834f39d","entry":"generate_stopword_stopping_criteria","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"supervised_generation.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/supervised_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50864f75b834f39d"}},{"code_sha256_prefix":"110e081edf72c07e","entry":"get_average_hidden_states","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_get_feature_X.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_get_feature_X.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"110e081edf72c07e"}},{"code_sha256_prefix":"5700274b87b272c4","entry":"get_index_of_valid_X","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_load_X_and_Y.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_load_X_and_Y.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5700274b87b272c4"}},{"code_sha256_prefix":"7e7fb553593d2c2b","entry":"get_last_token_hidden_states","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_get_feature_X.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_get_feature_X.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e7fb553593d2c2b"}},{"code_sha256_prefix":"ceccab32110fb3d9","entry":"lasso_select_k_features","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_feature_selection.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_feature_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ceccab32110fb3d9"}},{"code_sha256_prefix":"31807cc05985187a","entry":"load_llama2","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_load_model.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_load_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31807cc05985187a"}},{"code_sha256_prefix":"5d966d349a113a7b","entry":"mutual_info_select_k_features","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_feature_selection.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_feature_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d966d349a113a7b"}},{"code_sha256_prefix":"e32f292f91eab5d8","entry":"normalize_text","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/data_entry.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/data_entry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e32f292f91eab5d8"}},{"code_sha256_prefix":"cb4222238d0d7761","entry":"coqa_formatter_hf","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/data_entry.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/data_entry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb4222238d0d7761"}},{"code_sha256_prefix":"6d2be7746bd82de0","entry":"get_prob_statistics","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_get_feature_X.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_get_feature_X.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d2be7746bd82de0"}},{"code_sha256_prefix":"dfae1827665ab944","entry":"load_MMLU_X_Y","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_load_X_and_Y.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_load_X_and_Y.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfae1827665ab944"}},{"code_sha256_prefix":"a366d95b77db279a","entry":"load_X_Y_with_phase","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_load_X_and_Y.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_load_X_and_Y.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a366d95b77db279a"}},{"code_sha256_prefix":"b578098555ce09be","entry":"load_gemma","repo":"LoveCatc/supervised-llm-uncertainty-estimation","repo_kind":"official","path":"utils/funs_load_model.py","file_url":"https://github.com/LoveCatc/supervised-llm-uncertainty-estimation/blob/HEAD/utils/funs_load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b578098555ce09be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}