Browse State-of-the-Art › Language Modelling

Language Modelling

7,012 papers with code · 55 benchmarks · 166 datasets archive 2025-07-28

MedicalMiscellaneousNatural Language Processing

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: Wikipedia

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

55 leaderboard tables shown for this task, 55 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 55 until expanded.

Syntology 64 newer papers were judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 5 placed): WikiText-103 (6 judged, 0 placed), enwik8 (1 judged, 0 placed), The Pile (1 judged, 0 placed), WikiText-2 (12 judged, 1 placed), LAMBADA (16 judged, 1 placed), Text8 (1 judged, 0 placed), OpenWebText (6 judged, 0 placed), C4 (27 judged, 3 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

DatasetBest model (first row in archive order)PaperCodeSyntologyCompare
WikiText-103 (89 rows) RETRO (7.5B) Improving language models by retrieving from trillions of tokens code Syntology 16 ran (of which 5 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 3 violated, 11 with no contract checked; 2 where Syntology's instrument failed) · 7 unverified Compare
Penn Treebank (Word Level) (43 rows) GPT-3 (Zero-Shot) Language Models are Few-Shot Learners code Syntology community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified Compare
enwik8 (42 rows) GPT-2 (48 layers, h=1600) Language Models are Unsupervised Multitask Learners code — Compare
The Pile (39 rows) Test-Time Fine-Tuning with SIFT + Llama-3.2 (3B) Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs code Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified Compare
WikiText-2 (38 rows) SparseGPT (175B, 50% Sparsity) SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot code Syntology official (archive's flag): 2 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified Compare
LAMBADA (37 rows) PaLM-540B (Few-Shot) PaLM: Scaling Language Modeling with Pathways code Syntology 32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified Compare
One Billion Word (27 rows) MDLM (AR baseline) Simple and Effective Masked Diffusion Language Models code Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified Compare
Text8 (24 rows) GPT-2 Language Models are Unsupervised Multitask Learners code — Compare
Penn Treebank (Character Level) (20 rows) Mogrifier LSTM + dynamic eval Mogrifier LSTM code — Compare
Hutter Prize (18 rows) Transformer-XL + RMS dynamic eval Dynamic Evaluation of Transformer Language Models code — Compare
OpenWebText (12 rows) MDLM-Prime Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking — — Compare
SALMon (10 rows) Spirit-LM (Expr.) Spirit LM: Interleaved Spoken and Written Language Model code Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified Compare
C4 (9 rows) Primer Primer: Searching for Efficient Transformers for Language Modeling code Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified Compare
BIG-bench-lite (3 rows) GLM-130B (3-shot) GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
Wiki-40B (3 rows) FLASH-Quad-8k Transformer Quality in Linear Time code — Compare
CLUE (AFQMC) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (C3) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (CMNLI) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (CMRC2018) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (DRCD) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (OCNLI_50K) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
CLUE (WSC1.1) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
FewCLUE (BUSTM) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
FewCLUE (CHID-FC) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
FewCLUE (CLUEWSC-FC) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
FewCLUE (EPRSTMT) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
FewCLUE (OCNLI-FC) (2 rows) GLM-130B GLM-130B: An Open Bilingual Pre-trained Model code Syntology official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified Compare
VietMed (2 rows) Hybrid 4-gram VietMed-Train + ExtraText VietMed: A Dataset and Benchmark for Automatic Speech Recognition... code — Compare
100 sleep nights of 8 caregivers (1 row) Gpt3 007: Democratically Finding The Cause of Packet Drops code — Compare
2000 HUB5 English (1 row) MMLU Spirit LM: Interleaved Spoken and Written Language Model code Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified Compare
A1 (1 row) Fdyf A Neural Algorithm of Artistic Style code Syntology 76 ran (of which 0 constructed an object rather than computing a result; 44 with no instrument failure: 7 honoured, 3 violated, 34 with no contract checked; 32 where Syntology's instrument failed) · 34 unverified Compare
Arxiv HEP-TH citation graph (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Bookcorpus2 (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Books3 (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Curation Corpus (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
DM Mathematics (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
enwik8 dev (1 row) Transformer-LS (small) Long-Short Transformer: Efficient Transformers for Language and Vision code Syntology official (archive's flag): 1 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified Compare
enwiki8 (1 row) PAR Transformer 24B Pay Attention when Required code — Compare
FreeLaw (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
GitHub (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Gutenberg PG-19 (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
HackerNews (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
language-modeling-recommendation (1 row) GPT2 Zero-Shot Recommendation as Language Modeling code — Compare
NIH ExPorter (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
OpenSubtitles (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
OpenWebtext2 (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
PhilPapers (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Pile CC (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
PTB Diagnostic ECG Database (1 row) I-DARTS Improved Differentiable Architecture Search for Language Modeling... code — Compare
PubMed Cognitive Control Abstracts (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
PubMed Central (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
StackExchange (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
Text8 dev (1 row) Transformer-LS (small) Long-Short Transformer: Efficient Transformers for Language and Vision code Syntology official (archive's flag): 1 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified Compare
Ubuntu IRC (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare
USPTO Backgrounds (1 row) Gopher Scaling Language Models: Methods, Analysis & Insights from Training Gopher code — Compare

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

166 datasets whose archive record lists this task, ordered by the archive's paper count. 30 shown of 166 until expanded.

IMDb Movie ReviewsPubmedWikiText-2Penn TreebankC4WikiText-103The PileBIG-benchBookCorpusLAMBADAPubMedQALAMAOpenSubtitlesOpenWebTextWiCClothoAISHELL-1Libri-LightXQuADHate SpeechLRAPAWS-XATOMICELI5OLIDPG-19Billion Word BenchmarkBLUEBLiMPSIQAWritingPromptsT-RExCC100CLUEE2EKP20kTweetEvalMetaQARealNewsSemantic ScholarCMRCOSCARTED-LIUMSciDocsC3JerichoemrQACMRC 2018DARTMLSUMEmotionLinesOCNLIRWCPeerReadSMM4HChIDTatoebaAdvGLUEArxiv HEP-TH citation graph2000 HUB5 EnglishWorldtreeWiki-40BMassiveTextMuseDataKPTimesSentiCapKELMNatural StoriesDo-Not-AnswerPTB Diagnostic ECG DatabaseText8CLOTHSenseval-2Taskmaster-1CASIA-HWDBBeerAdvocateCMU DoGHumicroeditBabyLMCoDrawDakshinaIndoNLU BenchmarkOpen-PlatypusOVAD benchmarkANTIQUEPersonalDialogFewCLUEHutter PrizeHousekeepCC-StoriesART DatasetComQAIndoSumSALMonPEYMATUT Sound Events 2017UKPWinogender SchemasWMT 2018 NewsDefinite Pronoun Resolution DatasetGINCTUT-SED Synthetic 2016arXiv-10CLUECorpus2020CoarseWSD-20MovieFIBWikiCREMWNLaMProcaWaCCoached Conversational Preference ElicitationLiputan6Tencent ML-ImagesCCPE-MCommitChronicleDatabricks Dolly 15kHLA-Chatirc-disentanglementRONECSLINGVietMedChrEnHPLT v2IndicNLP CorpusKMIRSLNETWikiConvertWikiText-TL-39CircaComparative Question CompletionCzech restaurant informationNQuADPubMed Cognitive Control AbstractsRTCS-TESTSARTSentimental LIARSMC Text CorpusSpadesStack ExchangeViText2SQLAlexa Point of ViewblbooksBooks3Curation CorpusDpgMedia2019Fake News Filipino DatasetFreeLawGlot500-cGlotSparseGlotStoryBookHALvestHiXSTestInstructOpenWikiKitelanguage-modeling-recommendationLenta Short SentencesLipogram-eNTextPentachromatic Cultural Palette DatasetPerformance Improving Code Edits (PIE)PhilPapersPolyNewsSGXSTestSVLDUSPTO BackgroundsVerified Smart Contracts

Subtasks archive 2025-07-28

6 subtasks in the archive's task tree.

Papers archive 2025-07-28

archive papers tagged: 17,610 · with a code link: 7,012 · where Syntology ran a sample: 2,428 (1,953 with a run with no instrument failure, 475 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (2,428 of 17,610 tagged: 1,953 with a run with no instrument failure, 475 where every run was a failure of Syntology's instrument)

Page 1 of 177: papers 1 to 100 of 17,610, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.

Syntology lines on 88 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections