{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/polyglot-ner-massive-multilingual-named","title":"POLYGLOT-NER: Massive Multilingual Named Entity Recognition","arxiv_id":"1410.3791","date":"2014-10-14","proceeding":null,"authors":["Rami Al-Rfou","Vivek Kulkarni","Bryan Perozzi","Steven Skiena"],"abstract":"The increasing diversity of languages used on the web introduces a new level\nof complexity to Information Retrieval (IR) systems. We can no longer assume\nthat textual content is written in one language or even the same language\nfamily. In this paper, we demonstrate how to build massive multilingual\nannotators with minimal human expertise and intervention. We describe a system\nthat builds Named Entity Recognition (NER) annotators for 40 major languages\nusing Wikipedia and Freebase. Our approach does not require NER human annotated\ndatasets or language specific resources like treebanks, parallel corpora, and\northographic rules. The novelty of approach lies therein - using only language\nagnostic techniques, while achieving competitive performance.\n  Our method learns distributed word representations (word embeddings) which\nencode semantic and syntactic features of words in each language. Then, we\nautomatically generate datasets from Wikipedia link structure and Freebase\nattributes. Finally, we apply two preprocessing stages (oversampling and exact\nsurface form matching) which do not require any linguistic expertise.\n  Our evaluation is two fold: First, we demonstrate the system performance on\nhuman annotated datasets. Second, for languages where no gold-standard\nbenchmarks are available, we propose a new method, distant evaluation, based on\nstatistical machine translation.","url_abs":"http://arxiv.org/abs/1410.3791v1","url_pdf":"http://arxiv.org/pdf/1410.3791v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"multilingual-named-entity-recognition","task_name":"Multilingual Named Entity Recognition"},{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[{"slug":"polyglot-ner","name":"Polyglot-NER","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1410.3791","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}