{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offcon-3-what-is-state-of-the-art-anyway","title":"OffCon$^3$: What is state of the art anyway?","arxiv_id":"2101.11331","date":"2021-01-27","proceeding":null,"authors":["Philip J. Ball","Stephen J. Roberts"],"abstract":"Two popular approaches to model-free continuous control tasks are SAC and TD3. At first glance these approaches seem rather different; SAC aims to solve the entropy-augmented MDP by minimising the KL-divergence between a stochastic proposal policy and a hypotheical energy-basd soft Q-function policy, whereas TD3 is derived from DPG, which uses a deterministic policy to perform policy gradient ascent along the value function. In reality, both approaches are remarkably similar, and belong to a family of approaches we call `Off-Policy Continuous Generalized Policy Iteration'. This illuminates their similar performance in most continuous control benchmarks, and indeed when hyperparameters are matched, their performance can be statistically indistinguishable. To further remove any difference due to implementation, we provide OffCon$^3$ (Off-Policy Continuous Control: Consolidated), a code base featuring state-of-the-art versions of both algorithms.","url_abs":"https://arxiv.org/abs/2101.11331v2","url_pdf":"https://arxiv.org/pdf/2101.11331v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offcon-3-what-is-state-of-the-art-anyway","repo_url":"https://github.com/fiorenza2/OffCon3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"clipped-double-q-learning","method_name":"Clipped Double Q-learning"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dpg","method_name":"DPG"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dilated-convolution","method_name":"Dilated Convolution"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sac","method_name":"SAC"},{"method_slug":"td3","method_name":"TD3"},{"method_slug":"target-policy-smoothing","method_name":"Target Policy Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2101.11331","atlas_url":"https://app.syntology.ai/?focus=2101.11331","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}