Papers › Efficient Exploration through Bayesian Deep Q-Networks

Efficient Exploration through Bayesian Deep Q-Networks

13 Feb 2018ICLR 2018 1arXiv:1802.04412archive 2025-07-28

Kamyar Azizzadenesheli, Animashree Anandkumar

We study reinforcement learning (RL) in high dimensional episodic Markov decision processes (MDP). We consider value-based RL when the optimal Q-value is a linear function of d-dimensional state-action feature representation. For instance, in deep-Q networks (DQN), the Q-value is a linear function of the feature representation layer (output layer). We propose two algorithms, one based on optimism, LINUCB, and another based on posterior sampling, LINPSRL. We guarantee frequentist and Bayesian regret upper bounds of O(d sqrt{T}) for these two algorithms, where T is the number of episodes. We extend these methods to deep RL and propose Bayesian deep Q-networks (BDQN), which uses an efficient Thompson sampling algorithm for high dimensional RL. We deploy the double DQN (DDQN) approach, and instead of learning the last layer of Q-network using linear regression, we use Bayesian linear regression, resulting in an approximated posterior over Q-function. This allows us to directly incorporate the uncertainty over the Q-function and deploy Thompson sampling on the learned posterior distribution resulting in efficient exploration/exploitation trade-off. We empirically study the behavior of BDQN on a wide range of Atari games. Since BDQN carries out more efficient exploration and exploitation, it is able to reach higher return substantially faster compared to DDQN.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

kazizzad/BDQN-MxNet-Gluon officialmentioned in papermxnet report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Atari Games Efficient Exploration Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling regression

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Convolution DQN Dense Connections Double DQN Double Q-learning Experience Replay Q-Learning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections