Introspective Adversarial Network

The Introspective Adversarial Network (IAN) is a hybridization of GANs and VAEs that leverages the power of the adversarial objective while maintaining the VAE’s efficient inference mechanism. It uses the discriminator of the GAN, D, as a feature extractor for an inference subnetwork, E, which is implemented as a fully-connected layer on top of the final convolutional layer of the discriminator. We infer latent values Z ∼E(X) = q(Z|X) for reconstruction and sample random values Z ∼p(Z) from a standard normal for random image generation using the generator network, G.

Three distinct loss functions are used:

• ℒ_(img), the L1 pixel-wise reconstruction loss, which is preferred to the L2 reconstruction loss for its higher average gradient.
• ℒ_(𝒻ℯ𝒶𝓉𝓊𝓇ℯ), the feature-wise reconstruction loss, evaluated as the L2 difference between the original and reconstruction in the space of the hidden layers of the discriminator.
• ℒ_(adv), the ternary adversarial loss, a modification of the adversarial loss that forces the discriminator to label a sample as real, generated, or reconstructed (as opposed to a binary real vs. generated label).

Including the VAE’s KL divergence between the inferred latents E(X) and the prior p(Z), the loss function for the generator and encoder network is thus:

ℒ_(E, G) = λ_(adv)ℒ_(G_(adv)) + λ_(img)ℒ_(img) + λ_(feature)ℒ_(feature) + D_(KL)(E(X) || p(Z))

Where the λ terms weight the relative importance of each loss. We set λ_(img) to 3 and leave the other terms at 1. The discriminator is updated solely using the ternary adversarial loss. During each training step, the generator produces reconstructions G(E(X)) (using the standard VAE reparameterization trick) from data X and random samples G(Z), while the discriminator observes X as well as the reconstructions and random samples, and both networks are simultaneously updated.