{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-model-based-pre-training-for","title":"Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels","arxiv_id":"2209.12016","date":"2022-09-24","proceeding":null,"authors":["Sai Rajeswar","Pietro Mazzaglia","Tim Verbelen","Alexandre Piché","Bart Dhoedt","Aaron Courville","Alexandre Lacoste"],"abstract":"Controlling artificial agents from visual sensory data is an arduous task. Reinforcement learning (RL) algorithms can succeed but require large amounts of interactions between the agent and the environment. To alleviate the issue, unsupervised RL proposes to employ self-supervised interaction and learning, for adapting faster to future tasks. Yet, as shown in the Unsupervised RL Benchmark (URLB; Laskin et al. 2021), whether current unsupervised strategies can improve generalization capabilities is still unclear, especially in visual control settings. In this work, we study the URLB and propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent, and a task-aware fine-tuning strategy combined with a new proposed hybrid planner, Dyna-MPC, to adapt the agent for downstream tasks. On URLB, our method obtains 93.59% overall normalized performance, surpassing previous baselines by a staggering margin. The approach is empirically evaluated through a large-scale empirical study, which we use to validate our design choices and analyze our models. We also show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation. Project website: https://masteringurlb.github.io/","url_abs":"https://arxiv.org/abs/2209.12016v2","url_pdf":"https://arxiv.org/pdf/2209.12016v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-model-based-pre-training-for","repo_url":"https://github.com/mazpie/mastering-urlb","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"unsupervised-reinforcement-learning","task_name":"Unsupervised Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.12016","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.12016"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mazpie/mastering-urlb","reach":null}],"summary":{"ran":14,"ran_draft_wrong":2,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":24,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3200c778b6d1d0a8","entry":"BernoulliDist","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3200c778b6d1d0a8"}},{"code_sha256_prefix":"acf159e783acb4b1","entry":"DistLayer","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acf159e783acb4b1"}},{"code_sha256_prefix":"478e69ff513053f9","entry":"Encoder","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"478e69ff513053f9"}},{"code_sha256_prefix":"1007de8c2714718e","entry":"EnsembleRSSM","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1007de8c2714718e"}},{"code_sha256_prefix":"3c568e9e0347391a","entry":"GRUCell","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c568e9e0347391a"}},{"code_sha256_prefix":"e2e72a75f391e079","entry":"NormLayer","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2e72a75f391e079"}},{"code_sha256_prefix":"66fbe95d6fa0d1a6","entry":"OneHotDist","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66fbe95d6fa0d1a6"}},{"code_sha256_prefix":"374c4448ac4f78c9","entry":"Optimizer","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"374c4448ac4f78c9"}},{"code_sha256_prefix":"eb6cf9a53f925599","entry":"RequiresGrad","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb6cf9a53f925599"}},{"code_sha256_prefix":"1ec2bdb7622b9c66","entry":"SampleDist","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ec2bdb7622b9c66"}},{"code_sha256_prefix":"bd0264844e671c00","entry":"SquashedNormal","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd0264844e671c00"}},{"code_sha256_prefix":"0e5160a9bd6bd23c","entry":"StreamNorm","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e5160a9bd6bd23c"}},{"code_sha256_prefix":"b88483dbb52ca5a8","entry":"TanhTransform","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b88483dbb52ca5a8"}},{"code_sha256_prefix":"58845dbc07016005","entry":"TruncatedNormal","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58845dbc07016005"}},{"code_sha256_prefix":"65b7e7497e68ef56","entry":"lambda_return","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65b7e7497e68ef56"}},{"code_sha256_prefix":"28027f3ba32cb402","entry":"static_scan_for_lambda_return","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28027f3ba32cb402"}},{"code_sha256_prefix":"17f4154f5359d704","entry":"stop_gradient","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17f4154f5359d704"}},{"code_sha256_prefix":"d0b227b9789fce4a","entry":"ActorCritic","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0b227b9789fce4a"}},{"code_sha256_prefix":"f4840712627e4756","entry":"Decoder","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4840712627e4756"}},{"code_sha256_prefix":"e6da38f0e6237df9","entry":"DreamerAgent","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6da38f0e6237df9"}},{"code_sha256_prefix":"9d13094ceaa00c11","entry":"MLP","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d13094ceaa00c11"}},{"code_sha256_prefix":"5fb819938bf3f347","entry":"WorldModel","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fb819938bf3f347"}},{"code_sha256_prefix":"c0833f1665786d11","entry":"hard_update_params","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0833f1665786d11"}},{"code_sha256_prefix":"c84e0c6f35ae5bd7","entry":"static_scan","repo":"mazpie/mastering-urlb","repo_kind":"official","path":"agent/dreamer.py","file_url":"https://github.com/mazpie/mastering-urlb/blob/HEAD/agent/dreamer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c84e0c6f35ae5bd7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}