{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llmaaa-making-large-language-models-as-active","title":"LLMaAA: Making Large Language Models as Active Annotators","arxiv_id":"2310.19596","date":"2023-10-30","proceeding":null,"authors":["Ruoyu Zhang","Yanzeng Li","Yongliang Ma","Ming Zhou","Lei Zou"],"abstract":"Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw k-NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights. Compared with previous approaches, LLMaAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines.","url_abs":"https://arxiv.org/abs/2310.19596v2","url_pdf":"https://arxiv.org/pdf/2310.19596v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llmaaa-making-large-language-models-as-active","repo_url":"https://github.com/ridiculouz/llmaaa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"dataset-generation","task_name":"Dataset Generation"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[{"method_slug":"k-nn","method_name":"k-NN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19596","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19596"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ridiculouz/llmaaa","reach":{"status":"ok"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"75018e1e84c7efa1","entry":"get_entities","repo":"ridiculouz/llmaaa","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/ridiculouz/llmaaa/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75018e1e84c7efa1"}},{"code_sha256_prefix":"77d4a84f015acfb1","entry":"ner_collate_fn","repo":"ridiculouz/llmaaa","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ridiculouz/llmaaa/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77d4a84f015acfb1"}},{"code_sha256_prefix":"720a30b93d1b6168","entry":"re_collate_fn","repo":"ridiculouz/llmaaa","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ridiculouz/llmaaa/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"720a30b93d1b6168"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}