Long-Context Understanding

Benchmarks archive 2025-07-28

5 leaderboard tables shown for this task, 5 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

4 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 54 papers with code (81 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  9 Jun 2023 11 repositories listed Syntology ran 0 of 12 samples · 12 unverified
  Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences.
• GPT-4 Technical Report
  15 Mar 2023 11 repositories listed Syntology ran 2 of 5 samples · 3 unverified · 1 pointer-only (licence)
  We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.
• GLM-130B: An Open Bilingual Pre-trained Model
  5 Oct 2022 9 repositories listed Syntology ran 5 of 21 samples · 16 unverified
  We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters.
• RULER: What's the Real Context Size of Your Long-Context Language Models?
  9 Apr 2024 6 repositories listed Syntology ran 5 of 5 samples · 0 unverified
  Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases.
• Gated Delta Networks: Improving Mamba2 with Delta Rule
  9 Dec 2024 4 repositories listed Syntology ran 5 of 13 samples · 8 unverified · 7 pointer-only (licence)
  Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited.
• CogVLM: Visual Expert for Pretrained Language Models
  6 Nov 2023 4 repositories listed
  We introduce CogVLM, a powerful open-source visual language foundation model.
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  11 May 2023 4 repositories listed
  Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence.
• FABLES: Evaluating faithfulness and content selection in book-length summarization
  1 Apr 2024 3 repositories listed Syntology ran 3 of 3 samples · 0 unverified
  While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially…
• InternLM2 Technical Report
  26 Mar 2024 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  28 Aug 2023 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding.
• L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
  3 Oct 2024 2 repositories listed
  Long-context models (LCMs) have made remarkable strides in recent years, offering users great convenience for handling tasks that involve long context, such as document summarization.
• Long-context LLMs Struggle with Long In-context Learning
  2 Apr 2024 2 repositories listed Syntology ran 7 of 7 samples · 0 unverified
  We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens.
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
  7 Nov 2023 2 repositories listed Syntology ran 1 of 3 samples · 2 unverified · 3 pointer-only (licence)
  Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks.
• S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
  23 Oct 2023 2 repositories listed Syntology ran 21 of 26 samples · 5 unverified · 26 pointer-only (licence)
  The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning.
• Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
  13 Jul 2025 1 repository listed
  Long-context language models (LCLMs) have exhibited impressive capabilities in long-context understanding tasks.
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
  20 Jun 2025 1 repository listed
  In this paper, we propose the *KV footprint* as a unified metric, which accounts for both the amount of KV entries stored and their lifespan in memory.
• DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
  6 Jun 2025 1 repository listed
  Long-context understanding is crucial for many NLP applications, yet transformers struggle with efficiency due to the quadratic complexity of self-attention.
• MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
  5 Jun 2025 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
• SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
  27 May 2025 1 repository listed
  Speculative decoding is a widely adopted technique for accelerating inference in large language models (LLMs), but its performance degrades on long inputs due to increased attention cost and reduced draft accuracy.
• MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
  26 May 2025 1 repository listed
  Long Context Understanding (LCU) is a critical area for exploration in current large language models (LLMs).
• Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
  26 May 2025 1 repository listed Syntology ran 4 of 4 samples · 0 unverified · 2 pointer-only (licence)
  Post-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment.
• Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels
  20 May 2025 1 repository listed
  Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult.
• LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams
  24 Apr 2025 1 repository listed
  Long-context understanding poses significant challenges in natural language processing, particularly for real-world dialogues characterized by speech-based elements, high redundancy, and uneven information density.
• LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  22 Apr 2025 1 repository listed
  State space models (SSMs) have emerged as an efficient alternative to Transformer models for language modeling, offering linear computational complexity and constant memory usage as context length increases.
• Kimi-VL Technical Report
  10 Apr 2025 1 repository listed
  We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while…
• CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
  14 Mar 2025 1 repository listed
  We introduce CURIE, a scientific long-Context Understanding, Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists…
• Token Weighting for Long-Range Language Modeling
  12 Mar 2025 1 repository listed
  We evaluate all methods on multiple long-context understanding tasks and show that non-uniform loss weights are helpful to improve the long-context abilities of LLMs.
• LongProLIP: A Probabilistic Vision-Language Model with Long Context Text
  11 Mar 2025 1 repository listed Syntology ran 7 of 16 samples · 9 unverified · 16 pointer-only (licence)
  Recently, Probabilistic Language-Image Pre-Training (ProLIP) has been proposed to tackle the multiplicity issue of vision-language (VL) tasks.
• Self-Taught Agentic Long Context Understanding
  21 Feb 2025 1 repository listed
  Answering complex, long-context questions remains a major challenge for large language models (LLMs) as it requires effective question clarifications and context retrieval.
• SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning
  19 Feb 2025 1 repository listed
  Evaluating large language models' (LLMs) long-context understanding capabilities remains challenging.

Syntology lines on 14 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.