Papers › Count-Based Exploration in Feature Space for Reinforcement Learning

Count-Based Exploration in Feature Space for Reinforcement Learning

25 Jun 2017arXiv:1706.08090archive 2025-07-28

Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, Marcus Hutter

We introduce a new count-based optimistic exploration algorithm for Reinforcement Learning (RL) that is feasible in environments with high-dimensional state-action spaces. The success of RL algorithms in these domains depends crucially on generalisation from limited training experience. Function approximation techniques enable RL agents to generalise in order to estimate the value of unvisited states, but at present few methods enable generalisation regarding uncertainty. This has prevented the combination of scalable RL algorithms with efficient exploration strategies that drive the agent to reduce its uncertainty. We present a new method for computing a generalised state visit-count, which allows the agent to estimate the uncertainty associated with any state. Our \phi-pseudocount achieves generalisation by exploiting same feature representation of the state space that is used for value function approximation. States that have less frequently observed features are deemed more uncertain. The \phi-Exploration-Bonus algorithm rewards the agent for exploring in feature space rather than in the untransformed state space. The method is simpler and less computationally expensive than some previous proposals, and achieves near state-of-the-art results on high-dimensional RL benchmarks.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

aslanides/aixijs official report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Atari Games Efficient Exploration Reinforcement Learning Reinforcement Learning (RL)reinforcement-learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Atari Games	Atari 2600 Freeway	Sarsa-ε	Score	29.9	#34 of 59	Archive leaderboard	report
Atari Games	Atari 2600 Freeway	Sarsa-φ-EB	Score	0.0	#58 of 59	Archive leaderboard	report
Atari Games	Atari 2600 Frostbite	Sarsa-φ-EB	Score	2770.1	#27 of 53	Archive leaderboard	report
Atari Games	Atari 2600 Frostbite	Sarsa-ε	Score	1394.3	#34 of 53	Archive leaderboard	report
Atari Games	Atari 2600 Montezuma's Revenge	Sarsa-φ-EB	Score	2745.4	#13 of 50	Archive leaderboard	report
Atari Games	Atari 2600 Montezuma's Revenge	Sarsa-ε	Score	399.5	#22 of 50	Archive leaderboard	report
Atari Games	Atari 2600 Q*Bert	Sarsa-φ-EB	Score	4111.8	#46 of 57	Archive leaderboard	report
Atari Games	Atari 2600 Q*Bert	Sarsa-ε	Score	3895.3	#47 of 57	Archive leaderboard	report
Atari Games	Atari 2600 Venture	Sarsa-φ-EB	Score	1169.2	#18 of 55	Archive leaderboard	report
Atari Games	Atari 2600 Venture	Sarsa-ε	Score	0.0	#51 of 55	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections