Generalized State-Dependent Exploration

Generalized State-Dependent Exploration, or gSDE, is an exploration method for reinforcement learning that uses more general features and re-sampling the noise periodically.

State-Dependent Exploration (SDE) is an intermediate solution for exploration that consists in adding noise as a function of the state sₜ, to the deterministic action μ(𝐬ₜ). At the beginning of an episode, the parameters θ_ϵ of that exploration function are drawn from a Gaussian distribution. The resulting action 𝐚ₜ is as follows:

𝐚ₜ=μ(𝐬ₜ ; θ_μ)+ϵ(𝐬ₜ ; θ_ϵ), θ_ϵ ∼𝒩(0, σ²)

This episode-based exploration is smoother and more consistent than the unstructured step-based exploration. Thus, during one episode, instead of oscillating around a mean value, the action a for a given state s will be the same.

In the case of a linear exploration function ϵ(𝐬 ; θ_ϵ)=θ_ϵ 𝐬, by operation on Gaussian distributions, Rückstieß et al. show that the action element 𝐚ⱼ is normally distributed:

π]ⱼ(𝐚ⱼ |𝐬) ∼𝒩(μⱼ(𝐬), σ̂ⱼ̂²)

where σ̂ is a diagonal matrix with elements σ̂ⱼ=√(∑ᵢ(σ_(i j) 𝐬ᵢ)²).

Because we know the policy distribution, we can obtain the derivative of the log-likelihood logπ(𝐚 |𝐬) with respect to the variance σ :

(∂logπ(𝐚 |𝐬))/(∂σ_(i j))=((𝐚ⱼ-μⱼ)²-σ̂ⱼ̂²)/σ̂ⱼ³ (𝐬ᵢ² σ_(i j))/σ̂ⱼ̂

This can be easily plugged into the likelihood ratio gradient estimator, which allows to adapt σ during training. SDE is therefore compatible with standard policy gradient methods, while addressing most shortcomings of the unstructured exploration.

For gSDE, two improvements are suggested:

1. We sample the parameters θ_ϵ of the exploration function every n steps instead of every episode.
2. Instead of the state s, we can in fact use any features. We chose policy features 𝐳_μ(𝐬 ; θ_(𝐳_μ)) (last layer before the deterministic output .μ(𝐬)=θ_μ 𝐳_μ(𝐬 ; θ_(𝐳_μ))) as input to the noise function ϵ(𝐬 ; θ_ϵ)=θ_ϵ 𝐳_μ(𝐬)