Language Identification

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

6 leaderboard tables shown for this task, 6 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

20 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 143 papers with code (794 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Scaling Speech Technology to 1,000+ Languages
  22 May 2023 4 repositories listed Syntology ran 3 of 3 samples · 0 unverified
  Expanding the language coverage of speech technology has the potential to improve access to information for many more people.
• SpeechBrain: A General-Purpose Speech Toolkit
  8 Jun 2021 4 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  SpeechBrain is an open-source and all-in-one speech toolkit.
• The WiLI benchmark dataset for written language identification
  23 Jan 2018 4 repositories listed
  This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification.
• GlotLID: Language Identification for Low-Resource Languages
  24 Oct 2023 3 repositories listed
  Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages.
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
  23 May 2025 2 repositories listed Syntology ran 1 of 4 samples · 3 unverified
  Despite these advancements, CosyVoice 2 exhibits limitations in language coverage, domain diversity, data volume, text formats, and post-training techniques.
• GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
  31 Oct 2024 2 repositories listed
  The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models.
• From N-grams to Pre-trained Multilingual Models For Language Identification
  11 Oct 2024 2 repositories listed
  In this paper, we investigate the use of N-gram models and Large Pre-trained Multilingual models for Language Identification (LID) across 11 South African languages.
• KInIT at SemEval-2024 Task 8: Fine-tuned LLMs for Multilingual Machine-Generated Text Detection
  21 Feb 2024 2 repositories listed
  SemEval-2024 Task 8 is focused on multigenerator, multidomain, and multilingual black-box machine-generated text detection.
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
  17 Nov 2021 2 repositories listed
  On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.
• Discriminating Between Similar Nordic Languages
  11 Dec 2020 2 repositories listed
  Automatic language identification is a challenging problem.
• AdelaideCyC at SemEval-2020 Task 12: Ensemble of Classifiers for Offensive Language Detection in Social Media
  1 Dec 2020 2 repositories listed
  This paper describes the systems our team (AdelaideCyC) has developed for SemEval Task 12 (OffensEval 2020) to detect offensive language in social media.
• CyberTronics at SemEval-2020 Task 12: Multilingual Offensive Language Identification over Social Media
  1 Dec 2020 2 repositories listed
  This task has been organized for several languages, e.
• VoxLingua107: a Dataset for Spoken Language Recognition
  25 Nov 2020 2 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  Speech activity detection and speaker diarization are used to extract segments from the videos that contain speech.
• Common Voice: A Massively-Multilingual Speech Corpus
  13 Dec 2019 2 repositories listed
  To our knowledge this is the largest audio corpus in the public domain for speech recognition, both in terms of number of hours and number of languages.
• Word-level Embeddings for Cross-Task Transfer Learning in Speech Processing
  22 Oct 2019 2 repositories listed
  Recent breakthroughs in deep learning often rely on representation learning and knowledge transfer.
• SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
  19 Mar 2019 2 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  We present the results and the main findings of SemEval-2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval).
• Universal Dependency Parsing for Hindi-English Code-switching
  16 Apr 2018 2 repositories listed
  We present a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and propose a neural stacking model for parsing that efficiently leverages part-of-speech tag and syntactic tree…
• Advancing Uto-Aztecan Language Technologies: A Case Study on the Endangered Comanche Language
  10 May 2025 1 repository listed
  The digital exclusion of endangered languages remains a critical challenge in NLP, limiting both linguistic research and revitalization efforts.
• KréyoLID From Language Identification Towards Language Mining
  9 Mar 2025 1 repository listed
  Automatic language identification is frequently framed as a multi-class classification problem.
• English Please: Evaluating Machine Translation with Large Language Models for Multilingual Bug Reports
  20 Feb 2025 1 repository listed
  In this study, we conduct the first comprehensive evaluation of machine translation (MT) performance on bug reports, analyzing the capabilities of DeepL, AWS Translate, and large language models such as ChatGPT, Claude,…
• Multi-label Scandinavian Language Identification (SLIDE)
  10 Feb 2025 1 repository listed
  Identifying closely related languages at sentence level is difficult, in particular because it is often impossible to assign a sentence to a single language.
• Is It Navajo? Accurate Language Detection in Endangered Athabaskan Languages
  27 Jan 2025 1 repository listed
  Endangered languages, such as Navajo - the most widely spoken Native American language - are significantly underrepresented in contemporary language technologies, exacerbating the challenges of their preservation and…
• Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
  10 Jan 2025 1 repository listed
  While recent multilingual automatic speech recognition models claim to support thousands of languages, ASR for low-resource languages remains highly unreliable due to limited bimodal speech and text training data.
• AfriHuBERT: A self-supervised speech representation model for African languages
  30 Sep 2024 1 repository listed
  In this work, we present AfriHuBERT, an extension of mHuBERT-147, a compact self-supervised learning (SSL) model pretrained on 147 languages.
• Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking
  27 Sep 2024 1 repository listed
  Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical…
• Language-Informed Beam Search Decoding for Multilingual Machine Translation
  11 Aug 2024 1 repository listed Syntology ran 4 of 4 samples · 0 unverified
  Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, including multilingual NMT where the target language is specified as an input.
• Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
  7 Aug 2024 1 repository listed
  We present Speech-MASSIVE, a multilingual Spoken Language Understanding (SLU) dataset comprising the speech counterpart for a portion of the MASSIVE textual corpus.
• Script-Agnostic Language Identification
  25 Jun 2024 1 repository listed
  Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets.
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios
  13 Jun 2024 1 repository listed
  Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks.
• MaskLID: Code-Switching Language Identification through Iterative Masking
  10 Jun 2024 1 repository listed
  This method uses the LID itself to identify the features that require masking and does not rely on any external resource.

Syntology lines on 6 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.