Papers › The WiLI benchmark dataset for written language identification

The WiLI benchmark dataset for written language identification

23 Jan 2018arXiv:1801.07779archive 2025-07-28

Martin Thoma

This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification. WiLI-2018 is a publicly available, free of charge dataset of short text extracts from Wikipedia. It contains 1000 paragraphs of 235 languages, totaling in 23500 paragraphs. WiLI is a classification dataset: Given an unknown paragraph written in one dominant language, it has to be decided which language it is.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

LauraRuis/LanguageIdentification mentioned on GitHubpytorchSyntology: not harvested report
birolkuyumcu/language_identification mentioned on GitHubtfSyntology: not harvested report
sergeisoly/Russian-Language-Identification mentioned on GitHubpytorchSyntology: not harvested report
tsimafeip/nn-hackathon mentioned on GitHubSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

General ClassificationLanguage Identification

Datasets

Introduced by this paper, per the archive.

WiLI-2018

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections