DVD-GAN

DVD-GAN is a generative adversarial network for video generation built upon the BigGAN architecture.

DVD-GAN uses two discriminators: a Spatial Discriminator 𝒟_S and a Temporal Discriminator 𝒟_T. 𝒟_S critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually. The temporal discriminator 𝒟_T must provide G with the learning signal to generate movement (not evaluated by 𝒟_S).

The input to G consists of a Gaussian latent noise z ∼N(0, I) and a learned linear embedding e(y) of the desired class y. Both inputs are 120-dimensional vectors. G starts by computing an affine transformation of [z; e(y)] to a [4, 4, ch₀]-shaped tensor. [z; e(y)] is used as the input to all class-conditional Batch Normalization layers throughout G. This is then treated as the input (at each frame we would like to generate) to a Convolutional GRU.

This RNN is unrolled once per frame. The output of this RNN is processed by two residual blocks. The time dimension is combined with the batch dimension here, so each frame proceeds through the blocks independently. The output of these blocks has width and height dimensions which are doubled (we skip upsampling in the first block). This is repeated a number of times, with the output of one RNN + residual group fed as the input to the next group, until the output tensors have the desired spatial dimensions.

The spatial discriminator 𝒟_S functions almost identically to BigGAN’s discriminator. A score is calculated for each of the uniformly sampled k frames (default k = 8) and the 𝒟_S output is the sum over per-frame scores. The temporal discriminator 𝒟_T has a similar architecture, but pre-processes the real or generated video with a 2 ×2 average-pooling downsampling function ϕ. Furthermore, the first two residual blocks of 𝒟_T are 3-D, where every convolution is replaced with a 3-D convolution with a kernel size of 3 ×3 ×3. The rest of the architecture follows BigGAN.