head2head

one-to-many mapping. Giveing input variables $\bold{x}$, and output variables $\boldsymbol{y}$, assume latent variabes $\bold{z}$ can generate $\bold{y}$ from $\bold{x}$, the probability graphical graphs like this $\bold{x} \rightarrow \bold{z} \rightarrow \bold{y}$.

if we use $q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})$ to approximate $p_{\boldsymbol{\theta}}(\boldsymbol{\bold{z}|\bold{x}})$

We want to maximize the log likelihood,

$$\begin{aligned} \log p_{\boldsymbol{\theta}}(\bold{y}|\bold{x}) &= \log \int p(\bold{y}, \bold{z}|\bold{x}) d\bold{z} \\ &= \log \int p(\bold{y}| \bold{z}, \bold{x}) p(\bold{z}| \bold{x}) d \bold{z} \\ &= \log \int q(\bold{z}|\bold{x}) \frac{p(\bold{y}| \bold{z}, \bold{x}) p(\bold{z}| \bold{x})}{q(\bold{z}|\bold{x})} d \bold{z} \\ &\geq \int q(\bold{z}|\bold{x}) \log \frac{p(\bold{y}| \bold{z}, \bold{x}) p(\bold{z}| \bold{x})}{q(\bold{z}|\bold{x})} d \bold{z}\\ &= \mathbb{E}_{ q(\bold{z}|\bold{x})}\Bigg[\log p(\bold{y}| \bold{z}, \bold{x}) - \log \frac{q(\bold{z}|\bold{x})}{p(\bold{z}| \bold{x})}\Bigg] \\ &= \mathbb{E}_{ q(\bold{z}|\bold{x})} [\log p(\bold{y}| \bold{z}, \bold{x})] - KL(q(\bold{z}|\bold{x}) || p(\bold{z}| \bold{x})) \end{aligned}$$

Because $\bold{y}$ is conditioned on $\bold{z}$, thus $p(\bold{y}| \bold{z}, \bold{x})=p(\bold{y}| \bold{z})$.

$$\log p(\bold{y}|\bold{x}) = \mathbb{E}_{ q(\bold{z}|\bold{x})} [\log p(\bold{y}| \bold{z})] - KL(q(\bold{z}|\bold{x}) || p(\bold{z}| \bold{x}))$$