Offline RL

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

7 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 310 papers with code (755 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Decision Transformer: Reinforcement Learning via Sequence Modeling
  2 Jun 2021 20 repositories listed Syntology ran 17 of 26 samples · 9 unverified · 6 pointer-only (licence)
  In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling.
• Conservative Q-Learning for Offline Reinforcement Learning
  8 Jun 2020 18 repositories listed Syntology ran 24 of 34 samples · 10 unverified · 5 pointer-only (licence)
  We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
• Offline Reinforcement Learning with Implicit Q-Learning
  12 Oct 2021 17 repositories listed Syntology ran 32 of 58 samples · 26 unverified · 22 pointer-only (licence)
  The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable,…
• Reformer: The Efficient Transformer
  13 Jan 2020 10 repositories listed Syntology ran 6 of 8 samples · 2 unverified
  Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences.
• A Minimalist Approach to Offline Reinforcement Learning
  12 Jun 2021 8 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data.
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  29 Jun 2020 8 repositories listed Syntology ran 3 of 8 samples · 5 unverified · 3 pointer-only (licence)
  Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences.
• Rethinking Attention with Performers
  30 Sep 2020 7 repositories listed Syntology ran 9 of 16 samples · 7 unverified · 6 pointer-only (licence)
  We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity,…
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  15 Apr 2020 7 repositories listed Syntology ran 4 of 12 samples · 8 unverified
  In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL.
• MOPO: Model-based Offline Policy Optimization
  27 May 2020 6 repositories listed Syntology ran 3 of 8 samples · 5 unverified
  We also characterize the trade-off between the gain and risk of leaving the support of the batch data.
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  4 Oct 2021 5 repositories listed Syntology ran 13 of 21 samples · 8 unverified · 6 pointer-only (licence)
  However, prior methods typically require accurate estimation of the behavior policy or sampling from OOD data points, which themselves can be a non-trivial problem.
• Critic Regularized Regression
  26 Jun 2020 5 repositories listed
  Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction.
• Acme: A Research Framework for Distributed Reinforcement Learning
  1 Jun 2020 5 repositories listed
  These implementations serve both as a validation of our design decisions as well as an important contribution to reproducibility in RL research.
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
  28 Mar 2023 4 repositories listed Syntology ran 1 of 3 samples · 2 unverified
  This gives a deeper understanding of why the in-sample learning paradigm works, i.
• The In-Sample Softmax for Offline Reinforcement Learning
  28 Feb 2023 4 repositories listed
  We highlight a simple fact: it is more straightforward to approximate an in-sample \emph{softmax} using only actions in the dataset.
• Extreme Q-Learning: MaxEnt RL without Entropy
  5 Jan 2023 4 repositories listed Syntology ran 8 of 13 samples · 5 unverified · 7 pointer-only (licence)
  Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy.
• COMBO: Conservative Offline Model-Based Policy Optimization
  16 Feb 2021 4 repositories listed Syntology ran 0 of 2 samples · 2 unverified
  We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model.
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
  5 Jun 2020 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  We propose a novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN) that can effectively optimize a policy offline using 10-20 times fewer data than prior works.
• Revisiting the Minimalist Approach to Offline Reinforcement Learning
  16 May 2023 3 repositories listed Syntology ran 7 of 16 samples · 9 unverified
  Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity.
• Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  9 Mar 2023 3 repositories listed
  Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being…
• Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  13 Oct 2022 3 repositories listed
  To make practical, we further devise an offline RL algorithm to approximately find the solution.
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
  12 Aug 2022 3 repositories listed Syntology ran 11 of 18 samples · 7 unverified · 10 pointer-only (licence)
  In our approach, we learn an action-value function and we add a term maximizing action-values into the training loss of the conditional diffusion model, which results in a loss that seeks optimal actions that are near…
• cosFormer: Rethinking Softmax in Attention
  17 Feb 2022 3 repositories listed
  As one of its core components, the softmax attention helps to capture long-range dependencies yet prohibits its scale-up due to the quadratic space and time complexity to the sequence length.
• Supported Policy Optimization for Offline Reinforcement Learning
  13 Feb 2022 3 repositories listed
  Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy.
• Adversarially Trained Actor Critic for Offline Reinforcement Learning
  5 Feb 2022 3 repositories listed Syntology ran 3 of 3 samples · 0 unverified
  We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism.
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  1 Feb 2021 3 repositories listed
  We evaluate existing offline RL algorithms on NeoRL and argue that the performance of a policy should also be compared with the deterministic version of the behavior policy, instead of the dataset reward.
• The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints
  27 Oct 2018 3 repositories listed
  We show that a mild relaxation of the task and workspace constraints implicit in existing object grasping datasets can cause neural network based grasping algorithms to fail on even a simple block stacking task when…
• A Clean Slate for Offline Reinforcement Learning
  15 Apr 2025 2 repositories listed
  We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets.
• Flow Q-Learning
  4 Feb 2025 2 repositories listed Syntology ran 5 of 7 samples · 2 unverified · 5 pointer-only (licence)
  We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data.
• Offline Reinforcement Learning for LLM Multi-Step Reasoning
  20 Dec 2024 2 repositories listed Syntology ran 2 of 3 samples · 1 unverified
  While Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is less suitable for multi-step reasoning tasks because (1) DPO relies on paired preference data, which is not…
• Q-value Regularized Transformer for Offline Reinforcement Learning
  27 May 2024 2 repositories listed Syntology ran 2 of 8 samples · 6 unverified
  Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Conditional Sequence Modeling (CSM), a paradigm that learns the action distribution based on history trajectory and target…

Syntology lines on 20 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.