{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pretraining-representations-for-data","title":"Pretraining Representations for Data-Efficient Reinforcement Learning","arxiv_id":"2106.04799","date":"2021-06-09","proceeding":"NeurIPS 2021 12","authors":["Max Schwarzer","Nitarshan Rajkumar","Michael Noukhovitch","Ankesh Anand","Laurent Charlin","Devon Hjelm","Philip Bachman","Aaron Courville"],"abstract":"Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. To encourage learning representations which capture diverse aspects of the underlying MDP, we employ a combination of latent dynamics modelling and unsupervised goal-conditioned RL. When limited to 100k steps of interaction on Atari games (equivalent to two hours of human experience), our approach significantly surpasses prior work combining offline representation pretraining with task-specific finetuning, and compares favourably with other pretraining methods that require orders of magnitude more data. Our approach shows particular promise when combined with larger models as well as more diverse, task-aligned observational data -- approaching human-level performance and data-efficiency on Atari in our best setting. We provide code associated with this work at https://github.com/mila-iqia/SGI.","url_abs":"https://arxiv.org/abs/2106.04799v1","url_pdf":"https://arxiv.org/pdf/2106.04799v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pretraining-representations-for-data","repo_url":"https://github.com/mila-iqia/SGI","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"atari-games-100k","task_name":"Atari Games 100k"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04799","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04799"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mila-iqia/SGI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":1,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"2444d9ba3dc707f5","entry":"grouper","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/offline_dataset.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/offline_dataset.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2444d9ba3dc707f5"}},{"code_sha256_prefix":"5b01aec46fcf1f84","entry":"calculate_returns","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/scrl.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/scrl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b01aec46fcf1f84"}},{"code_sha256_prefix":"61a60f9cfd573aa4","entry":"delete_ind_from_array","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/rlpyt_utils.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/rlpyt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61a60f9cfd573aa4"}},{"code_sha256_prefix":"8e94ba500fb8f42e","entry":"delete_ind_from_tensor","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/rlpyt_utils.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/rlpyt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e94ba500fb8f42e"}},{"code_sha256_prefix":"92193540e742439a","entry":"discount_return_n_step","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/efficient_sampling_offline_dataset.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/efficient_sampling_offline_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92193540e742439a"}},{"code_sha256_prefix":"4050f8e6605a852d","entry":"exp_distance","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/scrl.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/scrl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4050f8e6605a852d"}},{"code_sha256_prefix":"9139d90ffb041630","entry":"get_augmentation","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9139d90ffb041630"}},{"code_sha256_prefix":"859eff6ebe1d7425","entry":"maybe_transform","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"859eff6ebe1d7425"}},{"code_sha256_prefix":"129a64cdda027059","entry":"norm_dist","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/scrl.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/scrl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"129a64cdda027059"}},{"code_sha256_prefix":"371be3f7fd66e816","entry":"sanitize_batch","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/rlpyt_buffer.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/rlpyt_buffer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"371be3f7fd66e816"}},{"code_sha256_prefix":"983a035f6286cecf","entry":"select_at_indexes","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"983a035f6286cecf"}},{"code_sha256_prefix":"067b295e116b1fe5","entry":"shuffle_batch_dim","repo":"mila-iqia/SGI","repo_kind":"official","path":"src/offline_dataset.py","file_url":"https://github.com/mila-iqia/SGI/blob/HEAD/src/offline_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"067b295e116b1fe5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}