{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-unconfident-llm-annotations-be-used-for","title":"Can Unconfident LLM Annotations Be Used for Confident Conclusions?","arxiv_id":"2408.15204","date":"2024-08-27","proceeding":null,"authors":["Kristina Gligorić","Tijana Zrnic","Cinoo Lee","Emmanuel J. Candès","Dan Jurafsky"],"abstract":"Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. In computational social science (CSS), researchers are increasingly leveraging LLM annotations to complement slow and expensive human annotations. Still, guidelines for collecting and using LLM annotations, without compromising the validity of downstream conclusions, remain limited. We introduce Confidence-Driven Inference: a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations should be collected, with the goal of producing accurate statistical estimates and provably valid confidence intervals while reducing the number of human annotations needed. Our approach comes with safeguards against LLM annotations of poor quality, guaranteeing that the conclusions will be both valid and no less accurate than if we only relied on human annotations. We demonstrate the effectiveness of Confidence-Driven Inference over baselines in statistical estimation tasks across three CSS settings--text politeness, stance, and bias--reducing the needed number of human annotations by over 25% in each. Although we use CSS settings for demonstration, Confidence-Driven Inference can be used to estimate most standard quantities across a broad range of NLP problems.","url_abs":"https://arxiv.org/abs/2408.15204v2","url_pdf":"https://arxiv.org/pdf/2408.15204v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-unconfident-llm-annotations-be-used-for","repo_url":"https://github.com/kristinagligoric/confidence-driven-inference","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.15204","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.15204"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kristinagligoric/confidence-driven-inference","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"79562521f7728505","entry":"lighten_color","repo":"kristinagligoric/confidence-driven-inference","repo_kind":"official","path":"code/plotting_utils.py","file_url":"https://github.com/kristinagligoric/confidence-driven-inference/blob/HEAD/code/plotting_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79562521f7728505"}},{"code_sha256_prefix":"3587d0b46af0a641","entry":"safe_expit","repo":"kristinagligoric/confidence-driven-inference","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/kristinagligoric/confidence-driven-inference/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3587d0b46af0a641"}},{"code_sha256_prefix":"e8da2c289e25c8ce","entry":"safe_log1pexp","repo":"kristinagligoric/confidence-driven-inference","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/kristinagligoric/confidence-driven-inference/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8da2c289e25c8ce"}},{"code_sha256_prefix":"4a9e92dc4104a6b0","entry":"train_tree","repo":"kristinagligoric/confidence-driven-inference","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/kristinagligoric/confidence-driven-inference/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a9e92dc4104a6b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}