Browse State-of-the-Art › Lexical Normalization

Lexical Normalization

16 papers with code · 1 benchmark · 1 dataset archive 2025-07-28

Natural Language Processing

Lexical normalization is the task of translating/transforming a non standard text to a standard register.

Example:

new pix comming tomoroe
new pictures coming tomorrow

Datasets usually consists of tweets, since these naturally contain a fair amount of these phenomena.

For lexical normalization, only replacements on the word-level are annotated. Some corpora include annotation for 1-N and N-1 replacements. However, word insertion/deletion and reordering is not part of the task.

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

DatasetBest model (first row in archive order)PaperCodeSyntologyCompare
LexNorm (4 rows) MoNoise MoNoise: Modeling Noise Using a Modular Normalization System code — Compare

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

1 dataset whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Papers archive 2025-07-28

archive papers tagged: 47 · with a code link: 16 · where Syntology ran a sample: 0 (0 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran: none of the 47 tagged papers has a Syntology run on record

Page 1 of 1: papers 1 to 47 of 47, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.

  • 10 Oct 2017 2 repositories listed
    We show that MoNoise beats the state-of-the-art on different normalization benchmarks for English and Dutch, which all define the task of normalization slightly different.
  • 13 Jan 2025 1 repository listed
    ViSoLex is an open-source system designed to address the unique challenges of lexical normalization for Vietnamese social media text.
  • 29 Jan 2024 1 repository listed
    In this work, we introduce Vietnamese Lexical Normalization (ViLexNorm), the first-ever corpus developed for the Vietnamese lexical normalization task.
  • 12 Nov 2023 1 repository listed
    Our dataset is accessible for research purposes.
  • 1 Oct 2022 1 repository listed
    Automatically detecting the intent of an utterance is important for various downstream natural language processing tasks.
  • 1 Nov 2021 1 repository listed
    This task is beneficial for downstream analysis, as it provides a way to harmonize (often spontaneous) linguistic variation.
  • 28 Oct 2021 1 repository listed
    We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 (van der Goot et al., 2021a), which evaluates lexical-normalization systems on 12 social media datasets in…
  • 24 May 2021 1 repository listed
    We examine language-specific versus multilingual BERT, and study the effect of lexical normalization on NER.
  • 8 Apr 2021 1 repository listed
    Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT).
  • 1 Apr 2021 1 repository listed
    Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of many natural language processing tasks on social media.
  • 31 Mar 2020 1 repository listed
    Roman Urdu is an informal form of the Urdu language written in Roman script, which is widely used in South Asia for online textual content.
  • 4 Jan 2020 1 repository listed
    Such informal and code-switched content are under-resourced in terms of labeled datasets and language models even for popular tasks like sentiment classification.
  • 29 Nov 2019 1 repository listed
    Our model achieves high accuracy for classification on this dataset and outperforms the previous model for multilingual text classification, highlighting language independence of McM.
  • 1 Jul 2019 1 repository listed
    In this paper, we introduce and demonstrate the online demo as well as the command line interface of a lexical normalization system (MoNoise) for a variety of languages.
  • 12 Apr 2019 1 repository listed
    Social media offer an abundant source of valuable raw data, however informal writing can quickly become a bottleneck for many natural language processing (NLP) tasks.
  • 1 Oct 2018 1 repository listed
    Recently introduced neural network parsers allow for new approaches to circumvent data sparsity issues by modeling character level information and by exploiting raw data in a semi-supervised setting.
  • A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media
    30 Sep 2024 0 repositories listed
    This study introduces an innovative automatic labeling framework to address the challenges of lexical normalization in social media texts for low-resource languages like Vietnamese.
  • A Character-level Ngram-based MT Approach for Lexical Normalization in Social Media
    17 Dec 2021 0 repositories listed
    This paper presents an ngram-based MT approach that operates at character-level to generate possible canonical forms for lexical variants in social media text.
  • A Text Editing Approach to Joint Japanese Word Segmentation, POS Tagging, and Lexical Normalization
    1 Nov 2021 0 repositories listed
    Lexical normalization, in addition to word segmentation and part-of-speech tagging, is a fundamental task for Japanese user-generated text processing.
  • CL-MoNoise: Cross-lingual Lexical Normalization
    1 Nov 2021 0 repositories listed
    Social media is notoriously difficult to process for existing natural language processing tools, because of spelling errors, non-standard words, shortenings, non-standard capitalization and punctuation.
  • Multilingual Sequence Labeling Approach to solve Lexical Normalization
    1 Nov 2021 0 repositories listed
    The task of converting a nonstandard text to a standard and readable text is known as lexical normalization.
  • Sesame Street to Mount Sinai: BERT-constrained character-level Moses models for multilingual lexical normalization
    1 Nov 2021 0 repositories listed
    This paper describes the HEL-LJU submissions to the MultiLexNorm shared task on multilingual lexical normalization.
  • To What Extent Does Lexical Normalization Help English-as-a-Second Language Learners to Read Noisy English Texts?
    1 Nov 2021 0 repositories listed
    How difficult is it for English-as-a-second language (ESL) learners to read noisy English texts?
  • Contrastive String Representation Learning using Synthetic Data
    8 Oct 2021 0 repositories listed
    String representation Learning (SRL) is an important task in the field of Natural Language Processing, but it remains under-explored.
  • Sequence-to-Sequence Lexical Normalization with Multilingual Transformers
    6 Oct 2021 0 repositories listed
    Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication.
  • Lexical Normalization for Code-switched Data and its Effect on POS-tagging
    1 Jun 2020 0 repositories listed
    Whereas monolingual social media data already provides many problems for natural language processing, CS adds another challenging dimension.
  • Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing
    1 May 2020 0 repositories listed
    Lexical normalization is the task of translating non-standard social media data to a standard form.
  • Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data?
    1 May 2020 0 repositories listed
    Social media is a valuable data resource for various natural language processing (NLP) tasks.
  • An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media
    1 Nov 2019 0 repositories listed
    Existing natural language processing systems have often been designed with standard texts in mind.
  • Enhancing BERT for Lexical Normalization
    1 Nov 2019 0 repositories listed
    Language model-based pre-trained representations have become ubiquitous in natural language processing.
  • Normalization of Indonesian-English Code-Mixed Twitter Data
    1 Nov 2019 0 repositories listed
  • Lexical Normalization of User-Generated Medical Text
    1 Aug 2019 0 repositories listed
  • Noise-Robust Morphological Disambiguation for Dialectal Arabic
    1 Jun 2018 0 repositories listed
  • A Taxonomy for In-depth Evaluation of Normalization for User Generated Content
    1 May 2018 0 repositories listed
  • Handling Normalization Issues for Part-of-Speech Tagging of Online Conversational Text
    1 May 2018 0 repositories listed
  • The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions
    1 May 2016 0 repositories listed
  • IHS_RD: Lexical Normalization for English Tweets
    1 Jul 2015 0 repositories listed
  • NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization
    1 Jul 2015 0 repositories listed
  • NCSU_SAS_SAM: Deep Encoding and Reconstruction for Normalization of Noisy Text
    1 Jul 2015 0 repositories listed
  • Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition
    1 Jul 2015 0 repositories listed
  • 1 Jul 2015 0 repositories listed
  • USZEGED: Correction Type-sensitive Normalization of English Tweets Using Efficiently Indexed n-gram Statistics
    1 Jul 2015 0 repositories listed
  • Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization
    1 Oct 2014 0 repositories listed
  • A Large Corpus of Product Reviews in Portuguese: Tackling Out-Of-Vocabulary Words
    1 May 2014 0 repositories listed
  • Towards Shared Datasets for Normalization Research
    1 May 2014 0 repositories listed
  • TweetNorm_es: an annotated corpus for Spanish microtext normalization
    1 May 2014 0 repositories listed
  • 1 Oct 2013 0 repositories listed

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections