Browse State-of-the-Art › Lemmatization
Lemmatization
68 papers with code · 0 benchmarks · 3 datasets archive 2025-07-28
Lemmatization is a process of determining a base or dictionary form (lemma) for a given surface form. Especially for languages with rich morphology it is important to be able to normalize words into their base forms to better support for example search engines and linguistic studies. Main difficulties in Lemmatization arise from encountering previously unseen words during inference time as well as disambiguating ambiguous surface forms which can be inflected variants of several different base forms depending on the context.
Source: Universal Lemmatizer: A Sequence to Sequence Model for Lemmatizing Universal Dependencies Treebanks
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
3 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 351 · with a code link: 68 · where Syntology ran a sample: 3 (2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (3 of 351 tagged: 2 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 4: papers 1 to 100 of 351, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
16 Mar 2020 5 repositories listed Syntology community repositories only · 16 ran (of which 8 constructed an object rather than computing a result; 10 with no instrument failure: 2 honoured, 0 violated, 8 with no contract checked; 6 where Syntology's instrument failed) · 14 unverified (of 30 harvested samples) · 28 pointer-only (licence)We introduce Stanza, an open-source Python natural language processing toolkit supporting 66 human languages.
-
24 Aug 2023 2 repositories listedThis paper presents a set of industrial-grade text processing models for Hungarian that achieve near state-of-the-art performance while balancing resource efficiency and accuracy.
-
24 Aug 2023 2 repositories listedIn this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text.
-
19 Aug 2020 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)Distributed representations of documents and words have gained popularity due to their ability to capture semantics of words and documents.
-
16 Mar 2019 2 repositories listedLemmatization of standard languages is concerned with (i) abstracting over morphological differences and (ii) resolving token-lemma ambiguities of inflected words in order to map them to a dictionary headword.
-
10 Aug 2018 2 repositories listedWe present LemmaTag, a featureless neural network architecture that jointly generates part-of-speech tags and lemmas for sentences by using bidirectional RNNs with character-level and word-level embeddings.
-
22 Mar 2025 1 repository listedThe study of historical languages presents unique challenges due to their complex orthographic systems, fragmentary textual evidence, and the absence of standardized digital representations of text in those languages.
-
15 Oct 2024 1 repository listedThis paper presents an experiment consisting in the comparison of six models to identify a state-of-the-art morphosyntactic parser and lemmatizer for Ancient Greek capable of annotating according to the Ancient Greek…
-
20 Sep 2024 1 repository listedMorphologically rich languages are notoriously challenging to process for downstream NLP applications.
-
18 Jun 2024 1 repository listedWe present an open-source web service for Czech morphosyntactic analysis.
-
30 May 2024 1 repository listedThis work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages.
-
31 Mar 2024 1 repository listedThe texts have been enriched with seven annotation layers: (i) tokenization layer; (ii) sentence segmentation layer; (iii) lemmatization layer; (iv) morphological layer; (v) dependency layer; (vi) dependency function…
-
30 Mar 2024 1 repository listedThe corpus consists of 5 017 documents on seven topics.
-
25 Mar 2024 1 repository listedWe experiment with seven languages of different morphological complexity, namely, English, Spanish, Basque, Russian, Czech, Turkish and Polish, using multilingual and language-specific pre-trained masked language…
-
6 Nov 2023 1 repository listedLemmatization holds significance in both natural language processing (NLP) and linguistics, as it effectively decreases data density and aids in comprehending contextual meaning.
-
3 Aug 2023 1 repository listedLemmatization is a Natural Language Processing (NLP) technique used to normalize text by changing morphological derivations of words to their root forms.
-
13 Jun 2023 1 repository listedLemmatization is still not a trivial task for morphologically rich languages.
-
23 May 2023 1 repository listedWhile prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical…
-
19 Apr 2023 1 repository listedAfter training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks.
-
3 Nov 2022 1 repository listedWhile transformer architectures with data augmentation achieved the most promising results for inflection and reinflection tasks, prefix-tuning on mGPT received the highest results for the analysis task.
-
5 Aug 2022 1 repository listedUnfortunately, at present, extraction of logical facts from unrestricted natural language is still too inaccurate to be used for reasoning, while restricting the grammar of the language (so-called controlled natural…
-
1 Jul 2022 1 repository listedWe introduce DadmaTools, an open-source Python Natural Language Processing toolkit for the Persian language.
-
5 Jun 2022 1 repository listedIn inflected languages, word endings play a prominent role, and hence different word forms cannot be recognized using generic text tokenization.
-
6 Jan 2022 1 repository listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)Although there are a couple of open-source language processing pipelines available for Hungarian, none of them satisfies the requirements of today's NLP applications.
-
8 Sep 2021 1 repository listedWe introduce ELIT, the Emory Language and Information Toolkit, which is a comprehensive NLP framework providing transformer-based end-to-end models for core tasks with a special focus on memory efficiency while…
-
7 Jul 2021 1 repository listedTexts written in Old Literary Finnish represent the first literary work ever written in Finnish starting from the 16th century.
-
26 May 2021 1 repository listedWe train neural models for morphological analysis, generation and lemmatization for morphologically rich languages.
-
28 Jan 2021 1 repository listedWe also compare with other methods of integrating external data into lemmatization and show that our enhanced system performs considerably better than a simple lexicon extension method based on the Stanza system, and it…
-
11 Jan 2021 1 repository listedIn the pipeline, one of the most critical and challenging problems is keyword mapping; constructing a mapping between tokens in the query and relational database elements (tables, attributes, values, etc.).
-
9 Jan 2021 1 repository listedFinally, we create a demo video for Trankit at: https://youtu.be/q0KGP3zGjGc.
-
8 Dec 2020 1 repository listed
-
1 Nov 2020 1 repository listed
-
28 Oct 2020 1 repository listed
-
30 Jul 2020 1 repository listed
-
17 Jun 2020 1 repository listed
-
21 May 2020 1 repository listed
-
18 May 2020 1 repository listed
-
2 Dec 2019 1 repository listed
-
1 Nov 2019 1 repository listed
-
24 Oct 2019 1 repository listed
-
22 Aug 2019 1 repository listed
-
1 Aug 2019 1 repository listed
-
1 Aug 2019 1 repository listed
-
23 Jul 2019 1 repository listed
-
1 Jun 2019 1 repository listed
-
1 Jun 2019 1 repository listed
-
29 May 2019 1 repository listed
-
2 Apr 2019 1 repository listed
-
1 Oct 2018 1 repository listed
-
1 Oct 2018 1 repository listed
-
1 Oct 2018 1 repository listed
-
1 Oct 2018 1 repository listed
-
10 Sep 2018 1 repository listed
-
31 Aug 2018 1 repository listed
-
1 Aug 2018 1 repository listed
-
1 Aug 2018 1 repository listed
-
26 Jul 2018 1 repository listed
-
1 Jun 2018 1 repository listed
-
1 May 2018 1 repository listed
-
1 May 2018 1 repository listed
-
6 Apr 2017 1 repository listed
-
1 Aug 2016 1 repository listed
-
1 Jun 2016 1 repository listed
-
1 May 2016 1 repository listed
-
1 May 2016 1 repository listed
-
4 Mar 2016 1 repository listed
-
24 May 2013 1 repository listed
-
14 Sep 2012 1 repository listed
-
NeoN: A Tool for Automated Detection, Linguistic and LLM-Driven Analysis of Neologisms in Polish21 May 2025 0 repositories listed
-
Breaking the Fake News Barrier: Deep Learning Approaches in Bangla Language30 Jan 2025 0 repositories listed
-
Context Aware Lemmatization and Morphological Tagging Method in Turkish4 Jan 2025 0 repositories listed
-
GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian29 Dec 2024 0 repositories listed
-
SinaTools: Open Source Toolkit for Arabic Natural Language Processing3 Nov 2024 0 repositories listed
-
A Comparative Study of Hybrid Models in Health Misinformation Text Classification8 Oct 2024 0 repositories listed
-
ACE-2005-PT: Corpus for Event Extraction in Portuguese29 Aug 2024 0 repositories listed
-
eFontes. Part of Speech Tagging and Lemmatization of Medieval Latin Texts.A Cross-Genre Survey29 Jun 2024 0 repositories listed
-
Recent advancements in computational morphology : A comprehensive survey8 Jun 2024 0 repositories listed
-
Joint Lemmatization and Morphological Tagging with LEMMING28 May 2024 0 repositories listed
-
Comparison of Current Approaches to Lemmatization: A Case Study in Estonian23 Apr 2024 0 repositories listed
-
TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages19 Apr 2024 0 repositories listed
-
ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus27 Mar 2024 0 repositories listed
-
The effect of stemming and lemmatization on Portuguese fake news text classification17 Oct 2023 0 repositories listed
-
Vacaspati: A Diverse Corpus of Bangla Literature11 Jul 2023 0 repositories listed
-
Advancing Full-Text Search Lemmatization Techniques with Paradigm Retrieval from OpenCorpora18 May 2023 0 repositories listed
-
LatinCy: Synthetic Trained Pipelines for Latin NLP7 May 2023 0 repositories listed
-
Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in Slavic Languages11 Apr 2023 0 repositories listed
-
On the Role of Morphological Information for Contextual Lemmatization1 Feb 2023 0 repositories listed
-
Automated Identification of Disaster News For Crisis Management Using Machine Learning24 Jan 2023 0 repositories listed
-
H2-Golden-Retriever: Methodology and Tool for an Evidence-Based Hydrogen Research Grantsmanship16 Nov 2022 0 repositories listed
-
Development of a rule-based lemmatization algorithm through Finite State Machine for Uzbek language28 Oct 2022 0 repositories listed
-
Arabic Word-level Readability Visualization for Assisted Text Simplification19 Oct 2022 0 repositories listed
-
Social Media Personal Event Notifier Using NLP and Machine Learning10 Oct 2022 0 repositories listed
-
Evaluating Diversity of Multiword Expressions in Annotated Text1 Oct 2022 0 repositories listed
-
Context based lemmatizer for Polish language23 Jul 2022 0 repositories listed
-
TArC: Tunisian Arabish Corpus First complete release11 Jul 2022 0 repositories listed
-
AGILe: The First Lemmatizer for Ancient Greek Inscriptions1 Jun 2022 0 repositories listed
-
An ELECTRA Model for Latin Token Tagging Tasks1 Jun 2022 0 repositories listed
-
Analyse Automatique de l’Ancien Arménien. Évaluation d’une méthode hybride « dictionnaire » et « réseau de neurones » sur un Extrait de l’Adversus Haereses d’Irénée de Lyon1 Jun 2022 0 repositories listed
-
Describing Language Variation in the Colophons of Armenian Manuscripts1 Jun 2022 0 repositories listed
-
Diachronic Parsing of Pre-Standard Irish1 Jun 2022 0 repositories listed
Syntology lines on 3 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections