{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dora-the-explorer-directed-outreaching","title":"DORA The Explorer: Directed Outreaching Reinforcement Action-Selection","arxiv_id":"1804.04012","date":"2018-04-11","proceeding":"ICLR 2018 1","authors":["Leshem Choshen","Lior Fox","Yonatan Loewenstein"],"abstract":"Exploration is a fundamental aspect of Reinforcement Learning, typically\nimplemented using stochastic action-selection. Exploration, however, can be\nmore efficient if directed toward gaining new world knowledge. Visit-counters\nhave been proven useful both in practice and in theory for directed\nexploration. However, a major limitation of counters is their locality. While\nthere are a few model-based solutions to this shortcoming, a model-free\napproach is still missing. We propose $E$-values, a generalization of counters\nthat can be used to evaluate the propagating exploratory value over\nstate-action trajectories. We compare our approach to commonly used RL\ntechniques, and show that using $E$-values improves learning and performance\nover traditional counters. We also show how our method can be implemented with\nfunction approximation to efficiently learn continuous MDPs. We demonstrate\nthis by showing that our approach surpasses state of the art performance in the\nFreeway Atari 2600 game.","url_abs":"http://arxiv.org/abs/1804.04012v1","url_pdf":"http://arxiv.org/pdf/1804.04012v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dora-the-explorer-directed-outreaching","repo_url":"https://github.com/borgr/DORA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"dora-the-explorer-directed-outreaching","repo_url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/dora","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.04012","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.04012"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/dora","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/borgr/DORA","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b407aa03e6b8a785","entry":"hashcoords","repo":"borgr/DORA","repo_kind":"official","path":"linapprox/tiles3.py","file_url":"https://github.com/borgr/DORA/blob/HEAD/linapprox/tiles3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b407aa03e6b8a785"}},{"code_sha256_prefix":"bf0c23acedec7379","entry":"tiles","repo":"borgr/DORA","repo_kind":"official","path":"linapprox/tiles3.py","file_url":"https://github.com/borgr/DORA/blob/HEAD/linapprox/tiles3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf0c23acedec7379"}},{"code_sha256_prefix":"2d3646df38ff4b06","entry":"tileswrap","repo":"borgr/DORA","repo_kind":"official","path":"linapprox/tiles3.py","file_url":"https://github.com/borgr/DORA/blob/HEAD/linapprox/tiles3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d3646df38ff4b06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}