Browse State-of-the-Art › Long-range modeling
Long-range modeling
69 papers with code · 2 benchmarks · 4 datasets archive 2025-07-28
A new task for testing the long-sequence modeling capabilities and efficiency of language models.
Image credit: SCROLLS: Standardized CompaRison Over Long Language Sequences
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 4 newer papers were judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): SCROLLS (1 judged, 0 placed), LRA (3 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| SCROLLS (13 rows) | CoLT5 XL | CoLT5: Faster Long-Range Transformers with Conditional Computation | — | Syntology 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified | Compare |
| LRA (7 rows) | S5 | Simplified State Space Layers for Sequence Modeling | code | Syntology official (archive's flag): 1 ran · 13 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
4 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 95 · with a code link: 69 · where Syntology ran a sample: 32 (28 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (32 of 95 tagged: 28 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 95 of 95, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
1 Dec 2023 35 repositories listed Syntology community repositories only · 28 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 0 honoured, 0 violated, 20 with no contract checked; 8 where Syntology's instrument failed) · 34 unverified (of 62 harvested samples) · 29 pointer-only (licence)Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.
-
31 Oct 2021 8 repositories listed Syntology official (archive's flag): 5 ran · 37 ran (of which 8 constructed an object rather than computing a result; 21 with no instrument failure: 2 honoured, 1 violated, 18 with no contract checked; 16 where Syntology's instrument failed) · 18 unverified (of 55 harvested samples) · 3 pointer-only (licence)A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies.
-
21 Sep 2022 7 repositories listed Syntology official (archive's flag): 3 ran · 12 ran (of which 6 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 13 harvested samples) · 11 pointer-only (licence)The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences.
-
9 Aug 2022 6 repositories listed Syntology official (archive's flag): 1 ran · 13 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified (of 19 harvested samples) · 6 pointer-only (licence)Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks.
-
8 Nov 2020 5 repositories listed Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
-
15 Dec 2021 4 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models.
-
9 Apr 2024 3 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches.
-
28 Dec 2022 3 repositories listed Syntology official (archive's flag): 1 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 8 unverified (of 15 harvested samples)First, we use synthetic language modeling tasks to understand the gap between SSMs and attention.
-
31 Mar 2020 3 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)Spatial-temporal graphs have been widely used by skeleton-based action recognition algorithms to model human action dynamics.
-
12 Mar 2024 2 repositories listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 4 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (of 11 harvested samples) · 11 pointer-only (licence)In this paper, we introduce a Large Kernel Vision Mamba U-shape Network, or LKM-UNet, for medical image segmentation.
-
25 Jan 2024 2 repositories listedCapturing voxel-wise spatial correspondence across distinct modalities is crucial for medical image analysis.
-
30 Nov 2023 2 repositories listedThe recent success of multiple neural architectures like CNNs, Transformers, and MLP-Mixers motivated us to look for similarities and differences between them.
-
12 May 2023 2 repositories listedAnd based on this attention, a network called T-former is designed for image inpainting.
-
8 Aug 2022 2 repositories listedWhile large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge.
-
21 Jul 2022 2 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 3 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples)Weakly Supervised Object Localization (WSOL), which aims to localize objects by only using image-level labels, has attracted much attention because of its low annotation cost in real applications.
-
23 Jun 2022 2 repositories listedOn the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on…
-
10 May 2022 2 repositories listed Syntology community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 16 harvested samples)Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
-
22 Apr 2022 2 repositories listedThe overall computing cost of the new building block is as low as O(N logN).
-
27 Mar 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)Modeling long range dependencies in sequential data is a fundamental step towards attaining human-level performance in many modalities such as text, vision, audio and video.
-
10 Jan 2022 2 repositories listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples)NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild.
-
15 Jul 2025 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 1 pointer-only (licence)Achieving equity in healthcare accessibility requires lightweight yet high-performance solutions for medical image segmentation, particularly in resource-limited settings.
-
6 Jul 2025 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 10 harvested samples)We present the first work demonstrating that a pure Mamba block can achieve efficient Dense Global Fusion, meanwhile guaranteeing top performance for camera-LiDAR multi-modal 3D object detection.
-
25 May 2025 1 repository listedMost publicly available medical segmentation datasets are only partially labeled, with annotations provided for a subset of anatomical structures.
-
22 May 2025 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 2 pointer-only (licence)Masked language models (MLMs) allow bidirectional understanding but are inefficient, as only masked tokens contribute to the loss per step.
-
16 May 2025 1 repository listedThe point cloud classification tasks face the dual challenge of efficiently extracting local geometric features while maintaining model complexity.
-
27 Mar 2025 1 repository listedVideo anomaly detection (VAD) methods are mostly CNN-based or Transformer-based, achieving impressive results, but the focus on detection accuracy often comes at the expense of inference speed.
-
vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognition27 Mar 2025 1 repository listedThe interplay of these components ensures that vGamba leverages the low computational demands of SSMs while maintaining the accuracy of attention mechanisms for modeling long-range dependencies in vision tasks.
-
9 Mar 2025 1 repository listedBy utilizing the ConvAttn module, we significantly reduce the reliance on self-attention and its involved memory-bound operations while maintaining the representational capability of transformers.
-
5 Jan 2025 1 repository listedMedical image segmentation is a critical task in medical imaging analysis.
-
17 Dec 2024 1 repository listedTowards this goal, we make the best design choice through extensive experiment settings from data curation to context window extending and utilizing: (1) we analyze data sources and length distributions to construct…
-
12 Nov 2024 1 repository listed
-
9 Oct 2024 1 repository listed
-
7 Aug 2024 1 repository listed
-
25 Jul 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
16 Jul 2024 1 repository listed
-
4 Jun 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
29 May 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples) · 2 pointer-only (licence)
-
24 May 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
5 May 2024 1 repository listed
-
3 Apr 2024 1 repository listed
-
6 Mar 2024 1 repository listed Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
5 Feb 2024 1 repository listed
-
28 Dec 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
15 Dec 2023 1 repository listed
-
3 Dec 2023 1 repository listed Syntology official (archive's flag): 12 ran · 12 ran (of which 2 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified (of 21 harvested samples) · 21 pointer-only (licence)
-
14 Sep 2023 1 repository listed
-
28 Jul 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified (of 8 harvested samples)
-
21 Jun 2023 1 repository listed
-
19 Jun 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples)
-
14 Jun 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
31 May 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
24 May 2023 1 repository listed
-
11 May 2023 1 repository listed
-
7 May 2023 1 repository listed
-
17 Oct 2022 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
14 Oct 2022 1 repository listed
-
28 Sep 2022 1 repository listed
-
26 Sep 2022 1 repository listed
-
21 Sep 2022 1 repository listed
-
6 Sep 2022 1 repository listed
-
7 Aug 2022 1 repository listed
-
1 Aug 2022 1 repository listed
-
24 Jun 2022 1 repository listed
-
12 Jun 2022 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
6 Jan 2022 1 repository listed
-
16 Sep 2021 1 repository listed
-
19 Jun 2021 1 repository listed
-
25 May 2021 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples)
-
18 Feb 2020 1 repository listed
-
Med-URWKV: Pure RWKV With ImageNet Pre-training For Medical Image Segmentation12 Jun 2025 0 repositories listed
-
M2Restore: Mixture-of-Experts-based Mamba-CNN Fusion Framework for All-in-One Image Restoration9 Jun 2025 0 repositories listed
-
ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration11 Apr 2025 0 repositories listed
-
An Uncertainty Principle for Linear Recurrent Neural Networks13 Feb 2025 0 repositories listed
-
Is Long Range Sequential Modeling Necessary For Colorectal Tumor Segmentation?10 Feb 2025 0 repositories listed
-
ReGNet: Reciprocal Space-Aware Long-Range Modeling for Crystalline Property Prediction4 Feb 2025 0 repositories listed
-
Exploring Historical Information for RGBE Visual Tracking with Mamba1 Jan 2025 0 repositories listed
-
MambaXCTrack: Mamba-based Tracker with SSM Cross-correlation and Motion Prompt for Ultrasound Needle Tracking13 Nov 2024 0 repositories listed
-
S7: Selective and Simplified State Space Layers for Sequence Modeling4 Oct 2024 0 repositories listed
-
Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion3 Sep 2024 0 repositories listed
-
Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network6 Aug 2024 0 repositories listed
-
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences12 Jun 2024 0 repositories listed
-
SMR: State Memory Replay for Long Sequence Modeling27 May 2024 0 repositories listed
-
MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space25 May 2024 0 repositories listed
-
Improving FHB Screening in Wheat Breeding Using an Efficient Transformer Model7 Aug 2023 0 repositories listed
-
AICT: An Adaptive Image Compression Transformer12 Jul 2023 0 repositories listed
-
24 May 2023 0 repositories listed
-
HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation10 Apr 2023 0 repositories listed
-
Token Transformer: Can class token help window-based transformer build better long-range interactions?11 Nov 2022 0 repositories listed
-
Pose Guided Human Image Synthesis with Partially Decoupled GAN7 Oct 2022 0 repositories listed
-
19 Jun 2022 0 repositories listed
-
Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions20 Sep 2021 0 repositories listed
-
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy17 Sep 2021 0 repositories listed
-
Gated Relational Graph Attention Networks1 Jan 2021 0 repositories listed
Syntology lines on 33 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections