{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leco-learnable-episodic-count-for-task","title":"LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward","arxiv_id":"2210.05409","date":"2022-10-11","proceeding":null,"authors":["DaeJin Jo","Sungwoong Kim","Daniel Wontae Nam","Taehwan Kwon","Seungeun Rho","Jongmin Kim","Donghoon Lee"],"abstract":"Episodic count has been widely used to design a simple yet effective intrinsic motivation for reinforcement learning with a sparse reward. However, the use of episodic count in a high-dimensional state space as well as over a long episode time requires a thorough state compression and fast hashing, which hinders rigorous exploitation of it in such hard and complex exploration environments. Moreover, the interference from task-irrelevant observations in the episodic count may cause its intrinsic motivation to overlook task-related important changes of states, and the novelty in an episodic manner can lead to repeatedly revisit the familiar states across episodes. In order to resolve these issues, in this paper, we propose a learnable hash-based episodic count, which we name LECO, that efficiently performs as a task-specific intrinsic reward in hard exploration problems. In particular, the proposed intrinsic reward consists of the episodic novelty and the task-specific modulation where the former employs a vector quantized variational autoencoder to automatically obtain the discrete state codes for fast counting while the latter regulates the episodic novelty by learning a modulator to optimize the task-specific extrinsic reward. The proposed LECO specifically enables the automatic transition from exploration to exploitation during reinforcement learning. We experimentally show that in contrast to the previous exploration methods LECO successfully solves hard exploration problems and also scales to large state spaces through the most difficult tasks in MiniGrid and DMLab environments.","url_abs":"https://arxiv.org/abs/2210.05409v1","url_pdf":"https://arxiv.org/pdf/2210.05409v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leco-learnable-episodic-count-for-task","repo_url":"https://github.com/kakaobrain/leco","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"efficient-exploration","task_name":"Efficient Exploration"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.05409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.05409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/leco","reach":null}],"summary":{"ran":1,"ran_draft_wrong":6,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"992553c7fdf8d531","entry":"AttrDict","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"992553c7fdf8d531"}},{"code_sha256_prefix":"d6ddabf253abd0c6","entry":"get_obs_shape","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6ddabf253abd0c6"}},{"code_sha256_prefix":"be8197cbc393893b","entry":"nonlinearity","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be8197cbc393893b"}},{"code_sha256_prefix":"5525927097201875","entry":"update_params_by_adam","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5525927097201875"}},{"code_sha256_prefix":"8a8aaef31d1b40ec","entry":"update_params_by_adamw","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a8aaef31d1b40ec"}},{"code_sha256_prefix":"a17d8608a786ddd1","entry":"update_params_by_rms","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a17d8608a786ddd1"}},{"code_sha256_prefix":"fa95d51044e04fc8","entry":"update_params_by_sgd","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa95d51044e04fc8"}},{"code_sha256_prefix":"e735ccc6ba1315d0","entry":"CellCounterWrapper","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e735ccc6ba1315d0"}},{"code_sha256_prefix":"273dee399d8182f7","entry":"LirPG","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"273dee399d8182f7"}},{"code_sha256_prefix":"04c8ad4bbc8ab86b","entry":"RND","repo":"kakaobrain/leco","repo_kind":"official","path":"sample_factory/algorithms/appo/intrinsic.py","file_url":"https://github.com/kakaobrain/leco/blob/HEAD/sample_factory/algorithms/appo/intrinsic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04c8ad4bbc8ab86b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}