{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/universalner-targeted-distillation-from-large","title":"UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition","arxiv_id":"2308.03279","date":"2023-08-07","proceeding":null,"authors":["Wenxuan Zhou","Sheng Zhang","Yu Gu","Muhao Chen","Hoifung Poon"],"abstract":"Large language models (LLMs) have demonstrated remarkable generalizability, such as understanding arbitrary entities and relations. Instruction tuning has proven effective for distilling LLMs into more cost-efficient models such as Alpaca and Vicuna. Yet such student models still trail the original LLMs by large margins in downstream applications. In this paper, we explore targeted distillation with mission-focused instruction tuning to train student models that can excel in a broad application class such as open information extraction. Using named entity recognition (NER) for case study, we show how ChatGPT can be distilled into much smaller UniversalNER models for open NER. For evaluation, we assemble the largest NER benchmark to date, comprising 43 datasets across 9 diverse domains such as biomedicine, programming, social media, law, finance. Without using any direct supervision, UniversalNER attains remarkable NER accuracy across tens of thousands of entity types, outperforming general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute F1 points in average. With a tiny fraction of parameters, UniversalNER not only acquires ChatGPT's capability in recognizing arbitrary entity types, but also outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably, UniversalNER even outperforms by a large margin state-of-the-art multi-task instruction-tuned systems such as InstructUIE, which uses supervised NER examples. We also conduct thorough ablation studies to assess the impact of various components in our distillation approach. We release the distillation recipe, data, and UniversalNER models to facilitate future research on targeted distillation.","url_abs":"https://arxiv.org/abs/2308.03279v2","url_pdf":"https://arxiv.org/pdf/2308.03279v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"universalner-targeted-distillation-from-large","repo_url":"https://github.com/emma1066/retrieval-augmented-it-openner","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"universalner-targeted-distillation-from-large","repo_url":"https://github.com/universal-ner/universal-ner","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"open-information-extraction","task_name":"Open Information Extraction"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/named-entity-recognition-on-ace-2005","task":"Named Entity Recognition (NER)","dataset":"ACE 2005","model":"UniNER-7B","rank_in_archive_order":8,"of":20,"metrics":{"F1":"86.69"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-anatem","task":"Named Entity Recognition (NER)","dataset":"AnatEM","model":"UniNER-7B","rank_in_archive_order":2,"of":5,"metrics":{"F1":"88.65"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc2gm","task":"Named Entity Recognition (NER)","dataset":"BC2GM","model":"UniNER-7B","rank_in_archive_order":13,"of":13,"metrics":{"F1":"82.42"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc4chemd","task":"Named Entity Recognition (NER)","dataset":"BC4CHEMD","model":"UniNER-7B","rank_in_archive_order":7,"of":7,"metrics":{"F1":"89.21"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-bc5cdr","task":"Named Entity Recognition (NER)","dataset":"BC5CDR","model":"UniNER-7B","rank_in_archive_order":10,"of":16,"metrics":{"F1":"89.34"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-conll03","task":"Named Entity Recognition (NER)","dataset":"CoNLL03","model":"UniNER-7B","rank_in_archive_order":1,"of":5,"metrics":{"F1":"93.3"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-findvehicle","task":"Named Entity Recognition (NER)","dataset":"FindVehicle","model":"UniNER-7B","rank_in_archive_order":3,"of":3,"metrics":{"F1":"98.3"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-genia","task":"Named Entity Recognition (NER)","dataset":"GENIA","model":"UniNER-7B","rank_in_archive_order":5,"of":14,"metrics":{"F1":"77.54"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI Disease","model":"UniNER-7B","rank_in_archive_order":1,"of":1,"metrics":{"F1":"86.96"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.03279","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.03279"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/universal-ner/universal-ner","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/emma1066/retrieval-augmented-it-openner","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran_violates":1,"ran":2,"unverified":7},"by_repo_kind":{"listed":{"samples":12,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a6a3ac30bf6e7bcf","entry":"gen_prompt","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/eval/eval_gpt_review.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/eval/eval_gpt_review.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6a3ac30bf6e7bcf"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/eval/evaluate.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/eval/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8049b382893c73dd","entry":"parse_score","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/eval/eval_gpt_review.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/eval/eval_gpt_review.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8049b382893c73dd"}},{"code_sha256_prefix":"7beeb01acc36919e","entry":"read_jsonl","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/eval/generate_webpage_data_from_table.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/eval/generate_webpage_data_from_table.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7beeb01acc36919e"}},{"code_sha256_prefix":"1d45bc90fd0cd90c","entry":"trim_hanging_lines","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/eval/generate_webpage_data_from_table.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/eval/generate_webpage_data_from_table.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d45bc90fd0cd90c"}},{"code_sha256_prefix":"dab937567e76e9e3","entry":"get_conv_template","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/conversation.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dab937567e76e9e3"}},{"code_sha256_prefix":"26ac62acef82f73c","entry":"get_eval","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/eval/eval_gpt_review.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/eval/eval_gpt_review.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26ac62acef82f73c"}},{"code_sha256_prefix":"2f3cb5d0dcc18502","entry":"get_gpu_memory","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/utils.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3cb5d0dcc18502"}},{"code_sha256_prefix":"fbd915d7d3d14137","entry":"get_response","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbd915d7d3d14137"}},{"code_sha256_prefix":"823ac03e72032fcb","entry":"parser","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/eval/evaluate.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/eval/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"823ac03e72032fcb"}},{"code_sha256_prefix":"278f9963b50f1622","entry":"preprocess_instance","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"278f9963b50f1622"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"universal-ner/universal-ner","repo_kind":"listed","path":"src/train/fastchat/utils.py","file_url":"https://github.com/universal-ner/universal-ner/blob/HEAD/src/train/fastchat/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}