{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stabilizing-deep-q-learning-with-convnets-and","title":"Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation","arxiv_id":"2107.00644","date":"2021-07-01","proceeding":"NeurIPS 2021 12","authors":["Nicklas Hansen","Hao Su","Xiaolong Wang"],"abstract":"While agents trained by Reinforcement Learning (RL) can solve increasingly challenging tasks directly from visual observations, generalizing learned skills to novel environments remains very challenging. Extensive use of data augmentation is a promising technique for improving generalization in RL, but it is often found to decrease sample efficiency and can even lead to divergence. In this paper, we investigate causes of instability when using data augmentation in common off-policy RL algorithms. We identify two problems, both rooted in high-variance Q-targets. Based on our findings, we propose a simple yet effective technique for stabilizing this class of algorithms under augmentation. We perform extensive empirical evaluation of image-based RL using both ConvNets and Vision Transformers (ViT) on a family of benchmarks based on DeepMind Control Suite, as well as in robotic manipulation tasks. Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals. We further show that our method scales to RL with ViT-based architectures, and that data augmentation may be especially important in this setting.","url_abs":"https://arxiv.org/abs/2107.00644v2","url_pdf":"https://arxiv.org/pdf/2107.00644v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stabilizing-deep-q-learning-with-convnets-and","repo_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"stabilizing-deep-q-learning-with-convnets-and","repo_url":"https://github.com/SVJayanthi/dmcontrol-generalization-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"stabilizing-deep-q-learning-with-convnets-and","repo_url":"https://github.com/nicklashansen/svea-vit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.00644","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.00644"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SVJayanthi/dmcontrol-generalization-benchmark","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nicklashansen/svea-vit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_fixture":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1},"listed":{"samples":14,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6c2c84b4ccb7f17b","entry":"cat","repo":"SVJayanthi/dmcontrol-generalization-benchmark","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/SVJayanthi/dmcontrol-generalization-benchmark/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c2c84b4ccb7f17b"}},{"code_sha256_prefix":"39eace7e2822504f","entry":"drop_path","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/layers/drop.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/layers/drop.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39eace7e2822504f"}},{"code_sha256_prefix":"daa7b3a0355eb7c3","entry":"gaussian_logprob","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/modules.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/modules.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daa7b3a0355eb7c3"}},{"code_sha256_prefix":"2df17488b927be0b","entry":"gaussian_logprob","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/algorithms/modules.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2df17488b927be0b"}},{"code_sha256_prefix":"1ac4e5bffc3dbbd3","entry":"make_dir","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ac4e5bffc3dbbd3"}},{"code_sha256_prefix":"5b2c72c30130d62a","entry":"make_divisible","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/layers/helpers.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/layers/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b2c72c30130d62a"}},{"code_sha256_prefix":"121cb336a1e1278a","entry":"squash","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/algorithms/modules.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"121cb336a1e1278a"}},{"code_sha256_prefix":"1387f0bafec9b149","entry":"trunc_normal_","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/algorithms/modules.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1387f0bafec9b149"}},{"code_sha256_prefix":"b25eb65554d61412","entry":"batch_from_obs","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/augmentations.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b25eb65554d61412"}},{"code_sha256_prefix":"09be0f6d9a709601","entry":"do_green_screen","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/env/wrappers.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/env/wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09be0f6d9a709601"}},{"code_sha256_prefix":"653d5a3aa2c473be","entry":"drop_block_2d","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/layers/drop.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/layers/drop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"653d5a3aa2c473be"}},{"code_sha256_prefix":"97aa7c942cfb7d3d","entry":"drop_block_fast_2d","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/layers/drop.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/layers/drop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97aa7c942cfb7d3d"}},{"code_sha256_prefix":"4cd7bffb72ec1909","entry":"listdir","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cd7bffb72ec1909"}},{"code_sha256_prefix":"5912a589c21e0e7f","entry":"load_config","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5912a589c21e0e7f"}},{"code_sha256_prefix":"0604211861b8f1e9","entry":"random_conv","repo":"nicklashansen/dmcontrol-generalization-benchmark","repo_kind":"official","path":"src/augmentations.py","file_url":"https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/HEAD/src/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0604211861b8f1e9"}},{"code_sha256_prefix":"e944e6e5e6f2648b","entry":"random_conv","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/augmentations.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e944e6e5e6f2648b"}},{"code_sha256_prefix":"3fa399e318655bf7","entry":"rgb_to_hsv","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/env/wrappers.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/env/wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fa399e318655bf7"}},{"code_sha256_prefix":"adedb876e87c2947","entry":"squash","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/algorithms/modules.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/algorithms/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adedb876e87c2947"}},{"code_sha256_prefix":"c5bfdbe07ab3b06d","entry":"transform_batch_obs","repo":"SVJayanthi/dmcontrol-generalization-benchmark","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/SVJayanthi/dmcontrol-generalization-benchmark/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5bfdbe07ab3b06d"}},{"code_sha256_prefix":"847b6ac61bd69609","entry":"vit_crop","repo":"nicklashansen/svea-vit","repo_kind":"listed","path":"src/augmentations.py","file_url":"https://github.com/nicklashansen/svea-vit/blob/HEAD/src/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"847b6ac61bd69609"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}