{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coannotating-uncertainty-guided-work","title":"CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation","arxiv_id":"2310.15638","date":"2023-10-24","proceeding":null,"authors":["Minzhi Li","Taiwei Shi","Caleb Ziems","Min-Yen Kan","Nancy F. Chen","Zhengyuan Liu","Diyi Yang"],"abstract":"Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.","url_abs":"https://arxiv.org/abs/2310.15638v1","url_pdf":"https://arxiv.org/pdf/2310.15638v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coannotating-uncertainty-guided-work","repo_url":"https://github.com/salt-nlp/coannotating","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-annotation","task_name":"text annotation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.15638","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.15638"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salt-nlp/coannotating","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/SALT-NLP/CoAnnotating","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"be735c0fcc114e9b","entry":"check","repo":"salt-nlp/coannotating","repo_kind":"official","path":"Expertise_Estimation.py","file_url":"https://github.com/salt-nlp/coannotating/blob/HEAD/Expertise_Estimation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be735c0fcc114e9b"}},{"code_sha256_prefix":"9a2f0de094eeef71","entry":"check4","repo":"salt-nlp/coannotating","repo_kind":"official","path":"Expertise_Estimation.py","file_url":"https://github.com/salt-nlp/coannotating/blob/HEAD/Expertise_Estimation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a2f0de094eeef71"}},{"code_sha256_prefix":"8bbb175a11f36019","entry":"check5","repo":"salt-nlp/coannotating","repo_kind":"official","path":"Expertise_Estimation.py","file_url":"https://github.com/salt-nlp/coannotating/blob/HEAD/Expertise_Estimation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bbb175a11f36019"}},{"code_sha256_prefix":"5cb041415ae2ab05","entry":"calculate_entropy","repo":"SALT-NLP/CoAnnotating","repo_kind":"official","path":"Expertise_Estimation.py","file_url":"https://github.com/SALT-NLP/CoAnnotating/blob/HEAD/Expertise_Estimation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cb041415ae2ab05"}},{"code_sha256_prefix":"9aed264338d2a39d","entry":"prompt","repo":"salt-nlp/coannotating","repo_kind":"official","path":"LLM_Inferencing.py","file_url":"https://github.com/salt-nlp/coannotating/blob/HEAD/LLM_Inferencing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9aed264338d2a39d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}