{"url":"/method/a3c","slug":"a3c","name":"A3C","full_name":"A3C","full_name_withheld":false,"description_markdown":"**A3C**, **Asynchronous Advantage Actor Critic**, is a policy gradient algorithm in reinforcement learning that maintains a policy $\\pi\\left(a\\_{t}\\mid{s}\\_{t}; \\theta\\right)$ and an estimate of the value\r\nfunction $V\\left(s\\_{t}; \\theta\\_{v}\\right)$. It operates in the forward view and uses a mix of $n$-step returns to update both the policy and the value-function. The policy and the value function are updated after every $t\\_{\\text{max}}$ actions or when a terminal state is reached. The update performed by the algorithm can be seen as $\\nabla\\_{\\theta{'}}\\log\\pi\\left(a\\_{t}\\mid{s\\_{t}}; \\theta{'}\\right)A\\left(s\\_{t}, a\\_{t}; \\theta, \\theta\\_{v}\\right)$ where $A\\left(s\\_{t}, a\\_{t}; \\theta, \\theta\\_{v}\\right)$ is an estimate of the advantage function given by:\r\n\r\n$$\\sum^{k-1}\\_{i=0}\\gamma^{i}r\\_{t+i} + \\gamma^{k}V\\left(s\\_{t+k}; \\theta\\_{v}\\right) - V\\left(s\\_{t}; \\theta\\_{v}\\right)$$\r\n\r\nwhere $k$ can vary from state to state and is upper-bounded by $t\\_{max}$.\r\n\r\nThe critics in A3C learn the value function while multiple actors are trained in parallel and get synced with global parameters every so often. The gradients are accumulated as part of training for stability - this is like parallelized stochastic gradient descent.\r\n\r\nNote that while the parameters $\\theta$ of the policy and $\\theta\\_{v}$ of the value function are shown as being separate for generality, we always share some of the parameters in practice. We typically use a convolutional neural network that has one [softmax](https://paperswithcode.com/method/softmax) output for the policy $\\pi\\left(a\\_{t}\\mid{s}\\_{t}; \\theta\\right)$ and one linear output for the value function $V\\left(s\\_{t}; \\theta\\_{v}\\right)$, with all non-output layers shared.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Asynchronous Methods for Deep Reinforcement Learning","paper":"/paper/asynchronous-methods-for-deep-reinforcement","first_author":"Volodymyr Mnih","n_authors":8,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/asynchronous-methods-for-deep-reinforcement"},"source":{"url":"http://arxiv.org/abs/1602.01783v2","title":"Asynchronous Methods for Deep Reinforcement Learning","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Policy Gradient Methods","url":"/methods/category/policy-gradient-methods","pwc_aliases":[]}],"n_papers_tagged":57,"archive_num_papers":57,"papers_newest_first":[{"paper":null,"title":"Detecting and Mitigating Reward Hacking in Reinforcement Learning Systems: A Comprehensive Empirical Study","date":"2025-07-08","arxiv_id":"2507.05619","n_code_links":0,"syntology":null},{"paper":null,"title":"Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach","date":"2025-05-07","arxiv_id":"2505.04148","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Task Scheduling for Microservices via A3C-Based Reinforcement Learning","date":"2025-05-01","arxiv_id":"2505.00299","n_code_links":0,"syntology":null},{"paper":null,"title":"Demand-Aware Beam Hopping and Power Allocation for Load Balancing in Digital Twin empowered LEO Satellite Networks","date":"2024-10-29","arxiv_id":"2411.08896","n_code_links":0,"syntology":null},{"paper":null,"title":"Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts","date":"2024-10-22","arxiv_id":"2410.19852","n_code_links":0,"syntology":null},{"paper":null,"title":"Physical Informed-Inspired Deep Reinforcement Learning Based Bi-Level Programming for Microgrid Scheduling","date":"2024-10-15","arxiv_id":"2410.11932","n_code_links":0,"syntology":null},{"paper":null,"title":"Criticality and Safety Margins for Reinforcement Learning","date":"2024-09-26","arxiv_id":"2409.18289","n_code_links":0,"syntology":null},{"paper":null,"title":"Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN","date":"2024-07-22","arxiv_id":"2407.15656","n_code_links":0,"syntology":null},{"paper":null,"title":"A Deep Reinforcement Learning Approach for Trading Optimization in the Forex Market with Multi-Agent Asynchronous Distribution","date":"2024-05-30","arxiv_id":"2405.19982","n_code_links":0,"syntology":null},{"paper":null,"title":"Sum Throughput Maximization in Multi-BD Symbiotic Radio NOMA Network Assisted by Active-STAR-RIS","date":"2024-01-16","arxiv_id":"2401.08301","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning Actions and Control of Focus of Attention with a Log-Polar-like Sensor","date":"2023-09-22","arxiv_id":"2309.12634","n_code_links":0,"syntology":null},{"paper":null,"title":"Safety Margins for Reinforcement Learning","date":"2023-07-25","arxiv_id":"2307.13642","n_code_links":0,"syntology":null},{"paper":"/paper/relu-to-the-rescue-improve-your-on-policy","title":"ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages","date":"2023-06-02","arxiv_id":"2306.01460","n_code_links":1,"syntology":null},{"paper":null,"title":"Double A3C: Deep Reinforcement Learning on OpenAI Gym Games","date":"2023-03-04","arxiv_id":"2303.02271","n_code_links":0,"syntology":null},{"paper":"/paper/solving-the-side-chain-packing-arrangement-of","title":"Reinforcement Learning for Molecular Dynamics Optimization: A Stochastic Pontryagin Maximum Principle Approach","date":"2022-12-06","arxiv_id":"2212.03320","n_code_links":1,"syntology":null},{"paper":null,"title":"Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image","date":"2022-10-12","arxiv_id":"2210.05891","n_code_links":0,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-for-a-two-echelon","title":"Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains","date":"2022-04-20","arxiv_id":"2204.09603","n_code_links":1,"syntology":null},{"paper":null,"title":"RL-CoSeg : A Novel Image Co-Segmentation Algorithm with Deep Reinforcement Learning","date":"2022-04-12","arxiv_id":"2204.05951","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning Reward Machines: A Study in Partially Observable Reinforcement Learning","date":"2021-12-17","arxiv_id":"2112.09477","n_code_links":0,"syntology":null},{"paper":null,"title":"Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning","date":"2021-03-06","arxiv_id":"2103.04067","n_code_links":0,"syntology":null},{"paper":null,"title":"A review of motion planning algorithms for intelligent robotics","date":"2021-02-04","arxiv_id":"2102.02376","n_code_links":0,"syntology":null},{"paper":null,"title":"Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup","date":"2020-12-31","arxiv_id":"2012.15511","n_code_links":0,"syntology":null},{"paper":"/paper/dynamic-scheduling-for-stochastic-edge-cloud","title":"Dynamic Scheduling for Stochastic Edge-Cloud Computing Environments using A3C learning and Residual Recurrent Neural Networks","date":"2020-09-01","arxiv_id":"2009.02186","n_code_links":1,"syntology":null},{"paper":null,"title":"Lagrangian Duality in Reinforcement Learning","date":"2020-07-20","arxiv_id":"2007.09998","n_code_links":0,"syntology":null},{"paper":"/paper/particle-based-adaptive-discretization-for","title":"PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network","date":"2020-03-16","arxiv_id":"2003.06959","n_code_links":1,"syntology":null},{"paper":"/paper/explore-and-exploit-with-heterotic-line","title":"Explore and Exploit with Heterotic Line Bundle Models","date":"2020-03-10","arxiv_id":"2003.04817","n_code_links":1,"syntology":null},{"paper":null,"title":"Fully Asynchronous Policy Evaluation in Distributed Reinforcement Learning over Networks","date":"2020-03-01","arxiv_id":"2003.00433","n_code_links":0,"syntology":null},{"paper":null,"title":"A Visual Communication Map for Multi-Agent Deep Reinforcement Learning","date":"2020-02-27","arxiv_id":"2002.11882","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Roundabout Insertion using Deep Reinforcement Learning","date":"2020-01-03","arxiv_id":"2001.00786","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Coordination among Multiple Traffic Intersections Using Multi-Agent Reinforcement Learning","date":"2019-12-09","arxiv_id":"1912.03851","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":40},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":40},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":40},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":26},{"task":"/task/atari-games","name":"Atari Games","papers":12},{"task":"/task/q-learning","name":"Q-Learning","papers":8},{"task":"/task/decision-making","name":"Decision Making","papers":5},{"task":"/task/scheduling","name":"Scheduling","papers":5},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":3},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":3},{"task":"/task/representation-learning","name":"Representation Learning","papers":3},{"task":null,"name":"CPU","papers":2},{"task":"/task/continuous-control","name":"Continuous Control","papers":2},{"task":null,"name":"GPU","papers":2},{"task":"/task/mujoco","name":"MuJoCo","papers":2},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":2},{"task":"/task/partially-observable-reinforcement-learning","name":"Partially Observable Reinforcement Learning","papers":2},{"task":"/task/problem-decomposition","name":"Problem Decomposition","papers":2},{"task":"/task/segmentation","name":"Segmentation","papers":2},{"task":"/task/continuous-control","name":"continuous-control","papers":2}],"tasks_shown":20,"n_tasks":61,"usage_by_year":[{"year":"2016","papers":2},{"year":"2017","papers":11},{"year":"2018","papers":5},{"year":"2019","papers":10},{"year":"2020","papers":8},{"year":"2021","papers":3},{"year":"2022","papers":4},{"year":"2023","papers":4},{"year":"2024","papers":7},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/a3c"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}