{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuous-deep-q-learning-with-model-based","title":"Continuous Deep Q-Learning with Model-based Acceleration","arxiv_id":"1603.00748","date":"2016-03-02","proceeding":null,"authors":["Shixiang Gu","Timothy Lillicrap","Ilya Sutskever","Sergey Levine"],"abstract":"Model-free reinforcement learning has been successfully applied to a range of\nchallenging problems, and has recently been extended to handle large neural\nnetwork policies and value functions. However, the sample complexity of\nmodel-free algorithms, particularly when using high-dimensional function\napproximators, tends to limit their applicability to physical systems. In this\npaper, we explore algorithms and representations to reduce the sample\ncomplexity of deep reinforcement learning for continuous control tasks. We\npropose two complementary techniques for improving the efficiency of such\nalgorithms. First, we derive a continuous variant of the Q-learning algorithm,\nwhich we call normalized adantage functions (NAF), as an alternative to the\nmore commonly used policy gradient and actor-critic methods. NAF representation\nallows us to apply Q-learning with experience replay to continuous tasks, and\nsubstantially improves performance on a set of simulated robotic control tasks.\nTo further improve the efficiency of our approach, we explore the use of\nlearned models for accelerating model-free reinforcement learning. We show that\niteratively refitted local linear models are especially effective for this, and\ndemonstrate substantially faster learning on domains where such models are\napplicable.","url_abs":"http://arxiv.org/abs/1603.00748v1","url_pdf":"http://arxiv.org/pdf/1603.00748v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/AndrewJWashington/protodriver","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/BY571/Normalized-Advantage-Function-NAF-","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/MathPhysSim/PER-NAF","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/Sheldonmao/Learning-To-walk","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/axnedergaard/normalized-advantage-functions","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/dpoulopoulos/drl_continuous_control","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/ikostrikov/pytorch-rl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"continuous-deep-q-learning-with-model-based","repo_url":"https://github.com/jakegrigsby/deep_control","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1603.00748","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1603.00748"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MathPhysSim/PER-NAF","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Sheldonmao/Learning-To-walk","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BY571/Normalized-Advantage-Function-NAF-","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dpoulopoulos/drl_continuous_control","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ikostrikov/pytorch-rl","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/axnedergaard/normalized-advantage-functions","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jakegrigsby/deep_control","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AndrewJWashington/protodriver","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e1cc84cc85b77b11","entry":"calculate_optical_flow","repo":"AndrewJWashington/protodriver","repo_kind":"listed","path":"protodriver/utils.py","file_url":"https://github.com/AndrewJWashington/protodriver/blob/HEAD/protodriver/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1cc84cc85b77b11"}},{"code_sha256_prefix":"a99c840526c78144","entry":"is_outlier","repo":"AndrewJWashington/protodriver","repo_kind":"listed","path":"protodriver/utils.py","file_url":"https://github.com/AndrewJWashington/protodriver/blob/HEAD/protodriver/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a99c840526c78144"}},{"code_sha256_prefix":"bfa31334a3b80f82","entry":"process_image","repo":"AndrewJWashington/protodriver","repo_kind":"listed","path":"protodriver/utils.py","file_url":"https://github.com/AndrewJWashington/protodriver/blob/HEAD/protodriver/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfa31334a3b80f82"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}