Browse State-of-the-Art › Lemmatization
Lemmatization
68 papers with code · 0 benchmarks · 3 datasets archive 2025-07-28
Lemmatization is a process of determining a base or dictionary form (lemma) for a given surface form. Especially for languages with rich morphology it is important to be able to normalize words into their base forms to better support for example search engines and linguistic studies. Main difficulties in Lemmatization arise from encountering previously unseen words during inference time as well as disambiguating ambiguous surface forms which can be inflected variants of several different base forms depending on the context.
Source: Universal Lemmatizer: A Sequence to Sequence Model for Lemmatizing Universal Dependencies Treebanks
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
3 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Most implemented papers archive 2025-07-28
30 shown of 68 papers with code (351 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.
-
16 Mar 2020 5 repositories listed Syntology ran 16 of 30 samples · 14 unverified · 28 pointer-only (licence)We introduce Stanza, an open-source Python natural language processing toolkit supporting 66 human languages.
-
24 Aug 2023 2 repositories listedThis paper presents a set of industrial-grade text processing models for Hungarian that achieve near state-of-the-art performance while balancing resource efficiency and accuracy.
-
24 Aug 2023 2 repositories listedIn this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text.
-
19 Aug 2020 2 repositories listed Syntology ran 2 of 2 samples · 0 unverifiedDistributed representations of documents and words have gained popularity due to their ability to capture semantics of words and documents.
-
16 Mar 2019 2 repositories listedLemmatization of standard languages is concerned with (i) abstracting over morphological differences and (ii) resolving token-lemma ambiguities of inflected words in order to map them to a dictionary headword.
-
10 Aug 2018 2 repositories listedWe present LemmaTag, a featureless neural network architecture that jointly generates part-of-speech tags and lemmas for sentences by using bidirectional RNNs with character-level and word-level embeddings.
-
22 Mar 2025 1 repository listedThe study of historical languages presents unique challenges due to their complex orthographic systems, fragmentary textual evidence, and the absence of standardized digital representations of text in those languages.
-
15 Oct 2024 1 repository listedThis paper presents an experiment consisting in the comparison of six models to identify a state-of-the-art morphosyntactic parser and lemmatizer for Ancient Greek capable of annotating according to the Ancient Greek…
-
20 Sep 2024 1 repository listedMorphologically rich languages are notoriously challenging to process for downstream NLP applications.
-
18 Jun 2024 1 repository listedWe present an open-source web service for Czech morphosyntactic analysis.
-
30 May 2024 1 repository listedThis work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages.
-
31 Mar 2024 1 repository listedThe texts have been enriched with seven annotation layers: (i) tokenization layer; (ii) sentence segmentation layer; (iii) lemmatization layer; (iv) morphological layer; (v) dependency layer; (vi) dependency function…
-
30 Mar 2024 1 repository listedThe corpus consists of 5 017 documents on seven topics.
-
25 Mar 2024 1 repository listedWe experiment with seven languages of different morphological complexity, namely, English, Spanish, Basque, Russian, Czech, Turkish and Polish, using multilingual and language-specific pre-trained masked language…
-
6 Nov 2023 1 repository listedLemmatization holds significance in both natural language processing (NLP) and linguistics, as it effectively decreases data density and aids in comprehending contextual meaning.
-
3 Aug 2023 1 repository listedLemmatization is a Natural Language Processing (NLP) technique used to normalize text by changing morphological derivations of words to their root forms.
-
13 Jun 2023 1 repository listedLemmatization is still not a trivial task for morphologically rich languages.
-
23 May 2023 1 repository listedWhile prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical…
-
19 Apr 2023 1 repository listedAfter training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks.
-
3 Nov 2022 1 repository listedWhile transformer architectures with data augmentation achieved the most promising results for inflection and reinflection tasks, prefix-tuning on mGPT received the highest results for the analysis task.
-
5 Aug 2022 1 repository listedUnfortunately, at present, extraction of logical facts from unrestricted natural language is still too inaccurate to be used for reasoning, while restricting the grammar of the language (so-called controlled natural…
-
1 Jul 2022 1 repository listedWe introduce DadmaTools, an open-source Python Natural Language Processing toolkit for the Persian language.
-
5 Jun 2022 1 repository listedIn inflected languages, word endings play a prominent role, and hence different word forms cannot be recognized using generic text tokenization.
-
6 Jan 2022 1 repository listed Syntology ran 0 of 2 samples · 2 unverifiedAlthough there are a couple of open-source language processing pipelines available for Hungarian, none of them satisfies the requirements of today's NLP applications.
-
8 Sep 2021 1 repository listedWe introduce ELIT, the Emory Language and Information Toolkit, which is a comprehensive NLP framework providing transformer-based end-to-end models for core tasks with a special focus on memory efficiency while…
-
7 Jul 2021 1 repository listedTexts written in Old Literary Finnish represent the first literary work ever written in Finnish starting from the 16th century.
-
26 May 2021 1 repository listedWe train neural models for morphological analysis, generation and lemmatization for morphologically rich languages.
-
28 Jan 2021 1 repository listedWe also compare with other methods of integrating external data into lemmatization and show that our enhanced system performs considerably better than a simple lexicon extension method based on the Stanza system, and it…
-
11 Jan 2021 1 repository listedIn the pipeline, one of the most critical and challenging problems is keyword mapping; constructing a mapping between tokens in the query and relational database elements (tables, attributes, values, etc.).
-
9 Jan 2021 1 repository listedFinally, we create a demo video for Trankit at: https://youtu.
Syntology lines on 3 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections