{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-sample-efficiency-in-model-free-1","title":"Improving Sample Efficiency in Model-Free Reinforcement Learning from Images","arxiv_id":"1910.01741","date":"2019-10-02","proceeding":null,"authors":["Denis Yarats","Amy Zhang","Ilya Kostrikov","Brandon Amos","Joelle Pineau","Rob Fergus"],"abstract":"Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.","url_abs":"https://arxiv.org/abs/1910.01741v3","url_pdf":"https://arxiv.org/pdf/1910.01741v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-sample-efficiency-in-model-free-1","repo_url":"https://github.com/denisyarats/pytorch_sac_ae","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"improving-sample-efficiency-in-model-free-1","repo_url":"https://github.com/KarlXing/RL-Visual-Continuous-Control","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improving-sample-efficiency-in-model-free-1","repo_url":"https://github.com/clearoboticslab/tsymrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"improving-sample-efficiency-in-model-free-1","repo_url":"https://github.com/ku2482/rljax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1910.01741","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.01741"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KarlXing/RL-Visual-Continuous-Control","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/denisyarats/pytorch_sac_ae","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ku2482/rljax","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clearoboticslab/tsymrl","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"daa7b3a0355eb7c3","entry":"gaussian_logprob","repo":"denisyarats/pytorch_sac_ae","repo_kind":"official","path":"sac_ae.py","file_url":"https://github.com/denisyarats/pytorch_sac_ae/blob/HEAD/sac_ae.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daa7b3a0355eb7c3"}},{"code_sha256_prefix":"42958997018c3581","entry":"make_dir","repo":"denisyarats/pytorch_sac_ae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/denisyarats/pytorch_sac_ae/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42958997018c3581"}},{"code_sha256_prefix":"63cc0b17d508c362","entry":"module_hash","repo":"denisyarats/pytorch_sac_ae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/denisyarats/pytorch_sac_ae/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63cc0b17d508c362"}},{"code_sha256_prefix":"975f23eeea267e55","entry":"preprocess_obs","repo":"KarlXing/RL-Visual-Continuous-Control","repo_kind":"listed","path":"src/agent/sac_ae.py","file_url":"https://github.com/KarlXing/RL-Visual-Continuous-Control/blob/HEAD/src/agent/sac_ae.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"975f23eeea267e55"}},{"code_sha256_prefix":"e4f442d1770eacd1","entry":"squash","repo":"denisyarats/pytorch_sac_ae","repo_kind":"official","path":"sac_ae.py","file_url":"https://github.com/denisyarats/pytorch_sac_ae/blob/HEAD/sac_ae.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f442d1770eacd1"}},{"code_sha256_prefix":"b41b083f7956cdf2","entry":"make_encoder","repo":"denisyarats/pytorch_sac_ae","repo_kind":"official","path":"encoder.py","file_url":"https://github.com/denisyarats/pytorch_sac_ae/blob/HEAD/encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b41b083f7956cdf2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}