Stanford CS236: Deep Generative Models I 2023 I Lecture 5 - VAEs

Latent Variable Models

• Latent variable models introduce unobserved random variables (Z) to capture unobserved factors of variation in the data.
• These models aim to model the joint probability distribution between observed variables (X) and latent variables (Z).
• Latent variable models offer advantages such as increased flexibility and the ability to extract latent features for representation learning.
• By conditioning on latent features, modeling the distribution of data points becomes easier as there is less variation to capture.
• Deep neural networks are used to model the conditional distribution of X given Z, with parameters depending on the latent variables through neural networks.
• The challenge lies in learning the parameters of the neural network since the latent variables are not observed during training.
• The function of Z is to represent latent variables that affect the distribution of X.
• In this model, X is not autoregressively generated, and P(x|Z) is a simple Gaussian distribution.
• The parameters of the Gaussian distribution for P(x|Z) are determined through a potentially complex nonlinear relationship with respect to Z.
• The individual conditional distributions P(x|Z) are typically modeled with simple distributions like Gaussians, but other distributions can be used as well.
• The functions used to model P(x|Z) are the same for every Z, but different prior distributions can be used for Z.
• The motivation for modeling P(x|Z) and P(X) is to make learning easier by clustering the data using the Z variables and to potentially gain insights into the latent factors of variation in the data.
• The number of latent variables is a hyperparameter that is determined through training.
• Sampling from the model is straightforward by first sampling Z from a Gaussian and then sampling X from the corresponding Gaussian defined by the mean and covariance predicted by the neural networks.

Mixture of Gaussians

• The mixture of Gaussians is a simple example of a latent variable model where Z is a categorical random variable that determines the mixture component, and P(x|Z) is a Gaussian distribution with different means and covariances for each mixture component.
• Mixture models can be useful for clustering data and can provide a better fit to data compared to a single Gaussian distribution.
• Unsupervised learning aims to discover meaningful structures in data, but it's not always clear what constitutes a good structure or clustering.
• Mixture models, such as a mixture of Gaussians, can be used for unsupervised learning by identifying the mixture components and clustering data points accordingly.
• However, mixture models may not perform well on tasks like image classification unless the number of mixture components is extremely large.
• An example of using a generative model on MNIST data shows that it can achieve reasonable clustering, but the clustering is not perfect and there are limitations to what can be discovered.

Variational Autoencoders (VAEs)

• Variational autoencoders (VAEs) are a powerful way of combining simple models to create more expressive generative models.
• VAEs use a continuous latent variable Z, which can take an infinite number of values, instead of a finite number of mixture components.
• The means and standard deviations of the Gaussian components in a VAE are determined by neural networks, providing more flexibility than a mixture of Gaussians with a lookup table.
• The sampling process in a VAE is similar to that of a mixture of Gaussians, involving sampling from the latent variable Z and then using neural networks to determine the parameters of the Gaussian distribution from which to sample.
• In a mixture of Gaussians (MoG), Z is continuous, allowing for smooth transitions between clusters.
• The mean of the latent representation C can be interpreted as an average representation of all the data points.
• The prior distribution for Z does not have to be uniform; it can be any simple distribution that allows for efficient sampling.
• In a Gaussian mixture model (GMM), there is no neural network; instead, a lookup table is used to map the latent variable Z to the parameters of the Gaussian distribution.
• The marginal distribution over X in a mixture of Gaussians is obtained by integrating over all possible values of Z.
• The dimensionality of Z is typically much lower than the dimensionality of X, allowing for dimensionality reduction.
• It is possible to incorporate more information into the prior distribution by using a more complex model, such as an autoregressive model.
• The number of components in a mixture of Gaussians is equal to the number of classes K.

Challenges in Learning Latent Variable Models

• The challenge in learning mixture models is that the latent variables are missing, requiring marginalization over all possible completions of the data.
• Evaluating the marginal probability over X requires integrating over all possible values of the latent variables Z, which can be intractable.
• If the Z variables can only take a finite number of values, the sum can be computed by brute force, but for continuous variables, an integral is required.
• Gradient computations are also expensive, making direct optimization challenging.
• One approach is to use Monte Carlo sampling to approximate the sum, by randomly sampling a small number of values for Z and using the sample average as an approximation.
• Sampling from a uniform distribution over Z is used because the sum is being converted into an expectation with respect to a uniform distribution, making the approximation tractable.
• However, this approach is not ideal because it does not take into account the actual distribution of Z.
• Uniformly sampling latent variables for completion is not effective because most completions would have low probability and high variance.
• A smarter weighting of selecting latent variables is needed to improve the model's performance.

Latent Variables and Disentangled Representations

• Latent variables (Z) are not necessarily meaningful features like hair or eye color, but they capture important factors of variation in the data.
• Z can be a vector of multiple values, allowing for the representation of multiple salient factors of variation.
• There are challenges in learning disentangled representations where latent variables have clear semantic meanings.
• If labels are available, semi-supervised learning can be used to steer latent variables towards desired semantic meanings.
• Important sampling is used to sample latent variables more efficiently by focusing on important completions.

Importance Sampling and the Evidence Lower Bound (ELBO)

• The choice of the proposal distribution Q for important sampling is crucial and can significantly affect the model's performance.
• The goal is to estimate the log marginal probability of a data point, which is intractable to compute directly.
• Importance sampling is used to estimate the log marginal probability by sampling from a distribution Q(Z|X) and then taking the log of the ratio of the joint probability of (X, Z) under the true distribution and the joint probability of (X, Z) under Q(Z|X).
• This estimator is unbiased, but it can be improved by using Jensen's inequality to derive a lower bound on the log marginal probability.
• The evidence lower bound (ELBO) is a lower bound on the log marginal probability that can be optimized instead of the true log marginal probability.
• The choice of Q(Z|X) controls how tight the ELBO is, and a good choice of Q(Z|X) can make the ELBO a very good approximation to the true log marginal probability.
• It is easier to derive a lower bound on the log marginal probability than an upper bound.
• The tightness of the bound on the quantity of interest can be quantified.
• The bound becomes tight when Q is chosen to be the conditional distribution of Z given X under the model.
• This optimal Q distribution is not easy to evaluate, which is why other methods are needed.
• The optimal way of inferring the latent variables is to use the true distribution.
• Inverting the neural network to find the likely inputs that would produce a given X is generally hard.
• The machinery for training a VAE involves optimizing both P and Q.