ELBO

Evidence lower bound (ELBO), variational lower bound:

$$\mathcal{L}_{\boldsymbol{\theta}, \boldsymbol{\phi}}(\bold{x}) = \mathbb{E}_{q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})} [\log p_{\boldsymbol{\theta}}(\bold{x,z}) - \log q_{\boldsymbol{\phi}}(\bold{z|x})]$$

也可写成其他形式，

$$\begin{aligned} \mathcal{L}_{\boldsymbol{\theta}, \boldsymbol{\phi}}(\bold{x}) &= \mathbb{E}_{q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})} [\log p_{\boldsymbol{\theta}} (\bold{x|z}) p_{\boldsymbol{\theta}}(\bold{z}) - \log q_{\boldsymbol{\phi}}(\bold{z|x})] \\ &= - \mathbb{E}_{q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})} \log \frac{q_{\boldsymbol{\phi}}(\bold{z|x})}{p_{\boldsymbol{\theta}}(\bold{z})} + \mathbb{E}_{q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})} \log p_{\boldsymbol{\theta}} (\bold{x|z}) \\ &= - D_{KL}(p_{\boldsymbol{\theta}}(\bold{z}|\bold{x})||p_{\boldsymbol{\theta}}(\bold{z})) + \mathbb{E}_{q_{\boldsymbol{\phi}}(\bold{z}|\bold{x})} \log p_{\boldsymbol{\theta}} (\bold{x|z}) \end{aligned}$$

我们想要微分并且优化 lower bound $\mathcal{L}_{\boldsymbol{\theta}, \boldsymbol{\phi}}(\bold{x})$ w.r.t. generative parameters $\boldsymbol{\theta}$ 和 variational parameters $\boldsymbol{\phi}$.

如何计算两个多元高斯分布之间的 KL divergence ref