Taylor Expansion Policy Optimization

TayPO, or Taylor Expansion Policy Optimization, refers to a set of algorithms that apply the k-th order Taylor expansions for policy optimization. This generalizes prior work, including TRPO as a special case. It can be thought of unifying ideas from trust-region policy optimization and off-policy corrections. Taylor expansions share high-level similarities with both trust region policy search and off-policy corrections. To get high-level intuitions of such similarities, consider a simple 1D example of Taylor expansions. Given a sufficiently smooth real-valued function on the real line f : ℝ →ℝ, the k-th order Taylor expansion of f(x) at x₀ is

fₖ(x) = f(x₀)+∑ᵏᵢ₌₁[f⁽ⁱ⁾(x₀)/i!](x−x₀)ⁱ

where f⁽ⁱ⁾(x₀) are the i-th order derivatives at x₀. First, a common feature shared by Taylor expansions and trust-region policy search is the inherent notion of a trust region constraint. Indeed, in order for convergence to take place, a trust-region constraint is required |x − x₀| < R(f, x₀)¹. Second, when using the truncation as an approximation to the original function f_K(x) ≈f(x), Taylor expansions satisfy the requirement of off-policy evaluations: evaluate target policy with behavior data. Indeed, to evaluate the truncation f_K(x) at any x (target policy), we only require the behavior policy "data" at x₀ (i.e., derivatives f⁽ⁱ⁾(x₀)).