Papers › #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

15 Nov 2016NeurIPS 2017 12arXiv:1611.04717archive 2025-07-28

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel

Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

clementbernardd/Count-Based-Exploration mentioned on GitHubpytorch report

nhynes/abc mentioned on GitHubpytorch report

uoe-agents/derl mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Atari Games Continuous Control Deep Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)continuous-control reinforcement-learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Atari Games	Atari 2600 Freeway	TRPO-hash	Score	34.0	#1 of 59	Archive leaderboard	report
Atari Games	Atari 2600 Frostbite	TRPO-hash	Score	5214.0	#15 of 53	Archive leaderboard	report
Atari Games	Atari 2600 Montezuma's Revenge	TRPO-hash	Score	75	#29 of 50	Archive leaderboard	report
Atari Games	Atari 2600 Venture	TRPO-hash	Score	445.0	#23 of 55	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections