generative model

• Discriminative Model: Learn a probability distribution $p(y|\bold{x})$
• Generative Model: Learn a probability distribution $p(\bold{x})$. Sample to generate new data
• Conditional Generative Model: Learn $p(\bold{x}|y)$. Generate new data conditioned on input labels

Example

$\bold{x}$ could be an image, $y$ is the image's class.

Notation

Probability density functions (PDFs) and probability mass functions (PMFs), also simply called probability distributions.

Taxonomy of Generative Models

Variational Autoencoders

Let $\bold{x}$ be a random sample from an unknown distribution $p(\bold{x})$. We also call $\bold{x}$ ​ an observed variable. We assume that there exists an unobserved (latent) variable $\bold{z}$ that generates $\bold{x}$.

How to generate new data

After training, sample new data like this：

• Sample $\bold{z}^{(i)}$ from prior $p_{\boldsymbol{\theta}}(\bold{z})$. e.g. Gaussian unconditoned on $\boldsymbol{\theta}$, $p_{\boldsymbol{\theta}}(\bold{z}) = \mathcal{N}(\bold{z};\bold{0}, \bold{I})$
• Sample $\bold{x}^{(i)}$ from conditional $p_{}(\bold{x}|\bold{z}^{(i)})$

How to train the model

Basic idea 1: maximize the likelihood of data

If we can observe $\bold{z}$ for each $\bold{x}$ during training, we can train this model by maxmizing the likelihood of each $\bold{x}$ conditioned on its observed $\bold{z}$. In other words, we can train a conditional generative model $p_{}(\bold{x}|\bold{z})$.

But we can't observe $\bold{z}$, we need to maximize the marginal likelihood:

$$p_{\boldsymbol{\theta}}(\bold{x})\,.$$

Basic idea 2:

• Lecture 19: Generative Models I - YouTube
• 498_FA2019_lecture19.pdf (umich.edu)