{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/behavior-from-the-void-unsupervised-active","title":"Behavior From the Void: Unsupervised Active Pre-Training","arxiv_id":"2103.04551","date":"2021-03-08","proceeding":"NeurIPS 2021 12","authors":["Hao liu","Pieter Abbeel"],"abstract":"We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.","url_abs":"https://arxiv.org/abs/2103.04551v4","url_pdf":"https://arxiv.org/pdf/2103.04551v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"behavior-from-the-void-unsupervised-active","repo_url":"https://github.com/rll-research/url_benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"},{"task_slug":"unsupervised-reinforcement-learning","task_name":"Unsupervised Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.04551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.04551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rll-research/url_benchmark","reach":null}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c5f58e3d61777f53","entry":"ICM","repo":"rll-research/url_benchmark","repo_kind":"official","path":"agent/icm_apt.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/agent/icm_apt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5f58e3d61777f53"}},{"code_sha256_prefix":"875a14ea71682fe8","entry":"DDPGAgent","repo":"rll-research/url_benchmark","repo_kind":"official","path":"agent/icm_apt.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/agent/icm_apt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"875a14ea71682fe8"}},{"code_sha256_prefix":"bf8b1f936aa93dd1","entry":"ICMAPTAgent","repo":"rll-research/url_benchmark","repo_kind":"official","path":"agent/icm_apt.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/agent/icm_apt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf8b1f936aa93dd1"}},{"code_sha256_prefix":"4ed18414f8adbb71","entry":"PBE","repo":"rll-research/url_benchmark","repo_kind":"official","path":"agent/icm_apt.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/agent/icm_apt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ed18414f8adbb71"}},{"code_sha256_prefix":"520703bc6d295272","entry":"RMS","repo":"rll-research/url_benchmark","repo_kind":"official","path":"agent/icm_apt.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/agent/icm_apt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"520703bc6d295272"}},{"code_sha256_prefix":"2a76dfe317adeb03","entry":"make_agent","repo":"rll-research/url_benchmark","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/rll-research/url_benchmark/blob/HEAD/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a76dfe317adeb03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}