Stanford CS236: Deep Generative Models I 2023 I Lecture 9 - Normalizing Flows

Generative Adversarial Networks (GANs)

• GANs use an adversarial process to train a generator and a discriminator network.
• The generator network aims to produce realistic samples that can fool the discriminator, while the discriminator network tries to distinguish between real data and samples generated by the generator.
• GANs offer more flexibility in defining the generative model family since the training objective is not based solely on maximum likelihood.
• GANs have the advantage of not requiring likelihood evaluations, making them flexible in choosing the generator architecture.
• Training GANs is challenging due to the Mini-Max optimization problem and convergence issues.

Two-Sample Test

• Two-sample tests are used to determine whether two groups of samples come from the same probability distribution.
• A test statistic is used to compare the two groups of samples, and a threshold is set to determine whether to reject the null hypothesis.
• Choosing a good test statistic is important to minimize type one and type two errors.
• Instead of handcrafting a test statistic to compare two probability distributions, we can train a classifier (discriminator) to distinguish between samples from the data distribution and samples from the model distribution.
• The loss of the classifier can be used as a test statistic, where a high loss indicates that the two distributions are different and a low loss indicates that they are similar.

GANs Training

• The generator is trained to fool the discriminator by minimizing the objective function V.
• The optimal discriminator is the conditional probability of a point x belonging to the positive class (real samples).
• The K Divergence is minimized when P(Theta) = P, but in practice, we cannot reach the global optimum.
• Among suboptimal models, it may be preferable to have one that cannot fool a discriminator over one with high compression.
• A likelihood-based model can be used as a discriminator, but it defeats the purpose of not needing access to the likelihood.