{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zipfian-environments-for-reinforcement","title":"Zipfian environments for Reinforcement Learning","arxiv_id":"2203.08222","date":"2022-03-15","proceeding":null,"authors":["Stephanie C. Y. Chan","Andrew K. Lampinen","Pierre H. Richemond","Felix Hill"],"abstract":"As humans and animals learn in the natural world, they encounter distributions of entities, situations and events that are far from uniform. Typically, a relatively small set of experiences are encountered frequently, while many important experiences occur only rarely. The highly-skewed, heavy-tailed nature of reality poses particular learning challenges that humans and animals have met by evolving specialised memory systems. By contrast, most popular RL environments and benchmarks involve approximately uniform variation of properties, objects, situations or tasks. How will RL algorithms perform in worlds (like ours) where the distribution of environment features is far less uniform? To explore this question, we develop three complementary RL environments where the agent's experience varies according to a Zipfian (discrete power law) distribution. On these benchmarks, we find that standard Deep RL architectures and algorithms acquire useful knowledge of common situations and tasks, but fail to adequately learn about rarer ones. To understand this failure better, we explore how different aspects of current approaches may be adjusted to help improve performance on rare events, and show that the RL objective function, the agent's memory system and self-supervised learning objectives can all influence an agent's ability to learn from uncommon experiences. Together, these results show that learning robustly from skewed experience is a critical challenge for applying Deep RL methods beyond simulations or laboratories, and our Zipfian environments provide a basis for measuring future progress towards this goal.","url_abs":"https://arxiv.org/abs/2203.08222v2","url_pdf":"https://arxiv.org/pdf/2203.08222v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zipfian-environments-for-reinforcement","repo_url":"https://github.com/deepmind/zipfian_environments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2203.08222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.08222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/zipfian_environments","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e31538ce23bfca8d","entry":"add_background_color_to_image","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/zipfs_gridworld.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/zipfs_gridworld.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e31538ce23bfca8d"}},{"code_sha256_prefix":"3f1580b7097d12fa","entry":"ensure_connected","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/map_building.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/map_building.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f1580b7097d12fa"}},{"code_sha256_prefix":"c2ee6ccbf4f97abe","entry":"get_neighbors","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/map_building.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/map_building.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2ee6ccbf4f97abe"}},{"code_sha256_prefix":"8522d8dde6f92350","entry":"manhattan_dist","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/map_building.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/map_building.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8522d8dde6f92350"}},{"code_sha256_prefix":"1bf10fad499e37b3","entry":"uniform_rare_dist","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/zipfs_gridworld_core.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/zipfs_gridworld_core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bf10fad499e37b3"}},{"code_sha256_prefix":"8819f358fb99e439","entry":"zipf_dist","repo":"deepmind/zipfian_environments","repo_kind":"official","path":"gridworld/zipfs_gridworld_core.py","file_url":"https://github.com/deepmind/zipfian_environments/blob/HEAD/gridworld/zipfs_gridworld_core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8819f358fb99e439"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}