{"url":"/method/gsde","slug":"gsde","name":"gSDE","full_name":"Generalized State-Dependent Exploration","full_name_withheld":false,"description_markdown":"**Generalized State-Dependent Exploration**, or **gSDE**, is an exploration method for reinforcement learning that uses more general features and re-sampling the noise periodically. \r\n\r\nState-Dependent Exploration (SDE) is an intermediate solution for exploration that consists in adding noise as a function of the state $s\\_{t}$, to the deterministic action $\\mu\\left(\\mathbf{s}\\_{t}\\right)$. At the beginning of an episode, the parameters $\\theta\\_{\\epsilon}$ of that exploration function are drawn from a Gaussian distribution. The resulting action $\\mathbf{a}\\_{t}$ is as follows:\r\n\r\n$$\r\n\\mathbf{a}\\_{t}=\\mu\\left(\\mathbf{s}\\_{t} ; \\theta\\_{\\mu}\\right)+\\epsilon\\left(\\mathbf{s}\\_{t} ; \\theta\\_{\\epsilon}\\right), \\quad \\theta\\_{\\epsilon} \\sim \\mathcal{N}\\left(0, \\sigma^{2}\\right)\r\n$$\r\n\r\nThis episode-based exploration is smoother and more consistent than the unstructured step-based exploration. Thus, during one episode, instead of oscillating around a mean value, the action a for a given state $s$ will be the same.\r\n\r\nIn the case of a linear exploration function $\\epsilon\\left(\\mathbf{s} ; \\theta\\_{\\epsilon}\\right)=\\theta\\_{\\epsilon} \\mathbf{s}$, by operation on Gaussian distributions, Rückstieß et al. show that the action element $\\mathbf{a}\\_{j}$ is normally distributed:\r\n\r\n$$\r\n\\pi]_{j}\\left(\\mathbf{a}\\_{j} \\mid \\mathbf{s}\\right) \\sim \\mathcal{N}\\left(\\mu\\_{j}(\\mathbf{s}), \\hat{\\sigma\\_{j}}^{2}\\right)\r\n$$\r\n\r\nwhere $\\hat{\\sigma}$ is a diagonal matrix with elements $\\hat{\\sigma}\\_{j}=\\sqrt{\\sum\\_{i}\\left(\\sigma\\_{i j} \\mathbf{s}\\_{i}\\right)^{2}}$.\r\n\r\nBecause we know the policy distribution, we can obtain the derivative of the log-likelihood $\\log \\pi(\\mathbf{a} \\mid \\mathbf{s})$ with respect to the variance $\\sigma$ :\r\n\r\n$$\r\n\\frac{\\partial \\log \\pi(\\mathbf{a} \\mid \\mathbf{s})}{\\partial \\sigma_{i j}}=\\frac{\\left(\\mathbf{a}\\_{j}-\\mu\\_{j}\\right)^{2}-\\hat{\\sigma\\_{j}}^{2}}{\\hat{\\sigma}\\_{j}^{3}} \\frac{\\mathbf{s}\\_{i}^{2} \\sigma\\_{i j}}{\\hat{\\sigma_{j}}}\r\n$$\r\n\r\nThis can be easily plugged into the likelihood ratio gradient estimator, which allows to adapt $\\sigma$ during training. SDE is therefore compatible with standard policy gradient methods, while addressing most shortcomings of the unstructured exploration.\r\n\r\nFor gSDE, two improvements are suggested:\r\n\r\n1. We sample the parameters $\\theta\\_{\\epsilon}$ of the exploration function every $n$ steps instead of every episode.\r\n2. Instead of the state s, we can in fact use any features. We chose policy features $\\mathbf{z}\\_{\\mu}\\left(\\mathbf{s} ; \\theta\\_{\\mathbf{z}\\_{\\mu}}\\right)$ (last layer before the deterministic output $\\left.\\mu(\\mathbf{s})=\\theta\\_{\\mu} \\mathbf{z}\\_{\\mu}\\left(\\mathbf{s} ; \\theta_{\\mathbf{z}\\_{\\mu}}\\right)\\right)$ as input to the noise function $\\epsilon\\left(\\mathbf{s} ; \\theta\\_{\\epsilon}\\right)=\\theta\\_{\\epsilon} \\mathbf{z}\\_{\\mu}(\\mathbf{s})$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Smooth Exploration for Robotic Reinforcement Learning","paper":"/paper/generalized-state-dependent-exploration-for","first_author":"Antonin Raffin","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/generalized-state-dependent-exploration-for"},"source":{"url":"https://arxiv.org/abs/2005.05719v2","title":"Smooth Exploration for Robotic Reinforcement Learning","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Exploration Strategies","url":"/methods/category/exploration-strategies","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/generalized-state-dependent-exploration-for","title":"Smooth Exploration for Robotic Reinforcement Learning","date":"2020-05-12","arxiv_id":"2005.05719","n_code_links":4,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/continuous-control","name":"Continuous Control","papers":1},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":1},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":1},{"task":"/task/continuous-control","name":"continuous-control","papers":1},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":1}],"tasks_shown":5,"n_tasks":5,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/gsde"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}