{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/climatex-do-llms-accurately-assess-human","title":"ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?","arxiv_id":"2311.17107","date":"2023-11-28","proceeding":null,"authors":["Romain Lacombe","Kerrie Wu","Eddie Dilworth"],"abstract":"Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.","url_abs":"https://arxiv.org/abs/2311.17107v1","url_pdf":"https://arxiv.org/pdf/2311.17107v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"climatex-do-llms-accurately-assess-human","repo_url":"https://github.com/rlacombe/climatex","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.17107","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.17107"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rlacombe/climatex","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f2095ce2b7e87881","entry":"compute_metrics","repo":"rlacombe/climatex","repo_kind":"official","path":"bert_classifier.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/bert_classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2095ce2b7e87881"}},{"code_sha256_prefix":"11c668fee86bc4a5","entry":"compute_metrics","repo":"rlacombe/climatex","repo_kind":"official","path":"roberta_classifier.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/roberta_classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11c668fee86bc4a5"}},{"code_sha256_prefix":"16b5ee5626d85e98","entry":"extract_confidence","repo":"rlacombe/climatex","repo_kind":"official","path":"utils/experiments.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/utils/experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16b5ee5626d85e98"}},{"code_sha256_prefix":"1e698c73c9291cd9","entry":"get_overall_accuracy","repo":"rlacombe/climatex","repo_kind":"official","path":"utils/analysis.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/utils/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e698c73c9291cd9"}},{"code_sha256_prefix":"dabf93afe33950c3","entry":"get_overall_bias","repo":"rlacombe/climatex","repo_kind":"official","path":"utils/analysis.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/utils/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dabf93afe33950c3"}},{"code_sha256_prefix":"8e7874b29a0b4553","entry":"get_slope","repo":"rlacombe/climatex","repo_kind":"official","path":"utils/analysis.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/utils/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e7874b29a0b4553"}},{"code_sha256_prefix":"1b46880d779a00b5","entry":"get_zero_shot_prompt","repo":"rlacombe/climatex","repo_kind":"official","path":"utils/experiments.py","file_url":"https://github.com/rlacombe/climatex/blob/HEAD/utils/experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b46880d779a00b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}