LAPGAN

A LAPGAN, or Laplacian Generative Adversarial Network, is a type of generative adversarial network that has a Laplacian pyramid representation. In the sampling procedure following training, we have a set of generative convnet models {G₀, …, G_K}, each of which captures the distribution of coefficients hₖ for natural images at a different level of the Laplacian pyramid. Sampling an image is akin to a reconstruction procedure, except that the generative models are used to produce the hₖ’s:

Ĩₖ = u(Ĩₖ₊₁) + h̃ₖ = u(Ĩₖ₊₁) + Gₖ(zₖ, u(Ĩₖ₊₁))

The recurrence starts by setting Ĩ_(K+1) = 0 and using the model at the final level G_K to generate a residual image Ĩ_K using noise vector z_K: Ĩ_K = G_K(z_K). Models at all levels except the final are conditional generative models that take an upsampled version of the current image Ĩₖ₊₁ as a conditioning variable, in addition to the noise vector zₖ.

The generative models {G₀, …, G_K} are trained using the CGAN approach at each level of the pyramid. Specifically, we construct a Laplacian pyramid from each training image I. At each level we make a stochastic choice (with equal probability) to either (i) construct the coefficients hₖ either using the standard Laplacian pyramid coefficient generation procedure or (ii) generate them using $G_{k}:

h̃ₖ = Gₖ(zₖ, u(Iₖ₊₁))

Here Gₖ is a convnet which uses a coarse scale version of the image lₖ = u(Iₖ₊₁) as an input, as well as noise vector zₖ. Dₖ takes as input hₖ or h̃ₖ, along with the low-pass image lₖ (which is explicitly added to hₖ or h̃ₖ before the first convolution layer), and predicts if the image was real or generated. At the final scale of the pyramid, the low frequency residual is sufficiently small that it can be directly modeled with a standard GAN: h̃_K = G_K(z_K) and D_K only has h_K or h̃_K as input.

Breaking the generation into successive refinements is the key idea. We give up any “global” notion of fidelity; an attempt is never made to train a network to discriminate between the output of a cascade and a real image and instead the focus is on making each step plausible.