{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/count-based-exploration-with-the-successor","title":"Count-Based Exploration with the Successor Representation","arxiv_id":"1807.11622","date":"2018-07-31","proceeding":"ICLR 2019 5","authors":["Marlos C. Machado","Marc G. Bellemare","Michael Bowling"],"abstract":"In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required. Our approach is based on the successor representation (SR), which was originally introduced as a representation defining state generalization by the similarity of successor states. Here we show that the norm of the SR, while it is being learned, can be used as a reward bonus to incentivize exploration. In order to better understand this transient behavior of the norm of the SR we introduce the substochastic successor representation (SSR) and we show that it implicitly counts the number of times each state (or feature) has been observed. We use this result to introduce an algorithm that performs as well as some theoretically sample-efficient approaches. Finally, we extend these ideas to a deep RL algorithm and show that it achieves state-of-the-art performance in Atari 2600 games when in a low sample-complexity regime.","url_abs":"https://arxiv.org/abs/1807.11622v4","url_pdf":"https://arxiv.org/pdf/1807.11622v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"count-based-exploration-with-the-successor","repo_url":"https://github.com/mcmachado/count_based_exploration_sr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"count-based-exploration-with-the-successor","repo_url":"https://github.com/bonniesjli/DQN_SR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"efficient-exploration","task_name":"Efficient Exploration"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-2600-freeway","task":"Atari Games","dataset":"Atari 2600 Freeway","model":"DQNMMCe","rank_in_archive_order":35,"of":59,"metrics":{"Score":"29.5"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-gravitar","task":"Atari Games","dataset":"Atari 2600 Gravitar","model":"DQNMMCe","rank_in_archive_order":22,"of":53,"metrics":{"Score":"1078.3"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-montezumas-revenge","task":"Atari Games","dataset":"Atari 2600 Montezuma's Revenge","model":"DQN+SR","rank_in_archive_order":19,"of":50,"metrics":{"Score":"1778.8"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-montezumas-revenge","task":"Atari Games","dataset":"Atari 2600 Montezuma's Revenge","model":"DQNMMCe+SR","rank_in_archive_order":20,"of":50,"metrics":{"Score":"1778.6"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-private-eye","task":"Atari Games","dataset":"Atari 2600 Private Eye","model":"DQNMMCe+SR","rank_in_archive_order":47,"of":52,"metrics":{"Score":"99.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-solaris","task":"Atari Games","dataset":"Atari 2600 Solaris","model":"DQNMMCe","rank_in_archive_order":20,"of":23,"metrics":{"Score":"2244.6"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-venture","task":"Atari Games","dataset":"Atari 2600 Venture","model":"DQNMMCe+SR","rank_in_archive_order":16,"of":55,"metrics":{"Score":"1241.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1807.11622","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1807.11622"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mcmachado/count_based_exploration_sr","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bonniesjli/DQN_SR","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e5da063f1d00602e","entry":"argmax","repo":"mcmachado/count_based_exploration_sr","repo_kind":"official","path":"tabular/utils.py","file_url":"https://github.com/mcmachado/count_based_exploration_sr/blob/HEAD/tabular/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5da063f1d00602e"}},{"code_sha256_prefix":"b8b8dc406619778f","entry":"operation","repo":"mcmachado/count_based_exploration_sr","repo_kind":"official","path":"function_approximation/common/decorators.py","file_url":"https://github.com/mcmachado/count_based_exploration_sr/blob/HEAD/function_approximation/common/decorators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8b8dc406619778f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}