Skip to main content

KL divergence of two Gaussian distributions

For a multivariate Guassian random variable x∼N(μ,Σ)\bold{x} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)，the probability density function is

p(x)=1(2π)n/2∣Σ∣1/2exp⁡(−12(x−μ)TΣ−1(x−μ))p(\bold{x}) = \frac{1}{(2\pi)^{n/2} |\Sigma|^{1/2}} \exp(-\frac{1}{2}(\bold{x} - \boldsymbol{\mu})^T \Sigma^{-1} (\bold{x} - \boldsymbol{\mu}))
  • μ\boldsymbol{\mu} is n-dimensional mean vector
  • Σ\Sigma is n×nn\times n covariance matrix, Σ=E[(x−μ)(x−μ)T]\Sigma = \mathbb{E}[(\bold{x} - \boldsymbol{\mu})(\bold{x} - \boldsymbol{\mu})^T]

Two multivariate Guassian distributions p(x)=N(μ1,Σ1)p(\bold{x}) = \mathcal{N}(\boldsymbol{\mu}_1, \Sigma_1) and q(x)=N(μ2,Σ2)q(\bold{x}) = \mathcal{N}(\boldsymbol{\mu}_2, \Sigma_2).

DKL(p(x)∣∣q(x))=Ep(x)[log⁡p(x)q(x)]=Ep(x)[log⁡p(x)−log⁡q(x)]=Ep(x)[−log⁡(2π)n/2−12log⁡∣Σ1∣−12(x−μ1)TΣ1−1(x−μ1)+log⁡(2π)n/2+12log⁡∣Σ2∣+12(x−μ2)TΣ2−1(x−μ2)]=12(log⁡∣Σ2∣∣Σ1∣−Ep(x)[(x−μ1)TΣ1−1(x−μ1)]+Ep(x)[(x−μ2)TΣ2−1(x−μ2)])=12(log⁡∣Σ2∣∣Σ1∣−Ep(x)[tr((x−μ1)TΣ1−1(x−μ1))]+Ep(x)[tr((x−μ2)TΣ2−1(x−μ2))])=12(log⁡∣Σ2∣∣Σ1∣−Ep(x)[tr(Σ1−1(x−μ1)(x−μ1)T)]+Ep(x)[tr(Σ2−1(x−μ2)(x−μ2)T)])=12(log⁡∣Σ2∣∣Σ1∣−tr(Σ1−1Ep(x)[(x−μ1)(x−μ1)T])+tr(Σ2−1Ep(x)[(x−μ2)(x−μ2)T]))=12(log⁡∣Σ2∣∣Σ1∣−tr(Σ1−1Σ1)+tr(Σ2−1Ep(x)[(xxT−2xμ2T+μ2μ2T)]))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1Ep(x)[(Σ1+2xμ1T−μ1μ1T−2xμ2T+μ2μ2T)]))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1(Σ1+2μ1μ1T−μ1μ1T−2μ1μ2T+μ2μ2T)))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1Σ1)+tr(Σ2−1(μ1μ1T−2μ1μ2T+μ2μ2T)))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1Σ1)+tr(Σ2−1(μ1−μ2)(μ1−μ2)T))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1Σ1)+tr((μ1−μ2)TΣ2−1(μ1−μ2)))=12(log⁡∣Σ2∣∣Σ1∣−n+tr(Σ2−1Σ1)+(μ1−μ2)TΣ2−1(μ1−μ2))\begin{aligned} D_{KL}(p(\bold{x}) || q(\bold{x})) &= \mathbb{E}_{p(\bold{x})} \Bigg[\log \frac{p(\bold{x})}{q(\bold{x})}\Bigg]\\ &= \mathbb{E}_{p(\bold{x})} \Big[\log p(\bold{x}) - \log q(\bold{x})\Big]\\ &= \mathbb{E}_{p(\bold{x})} \Big[- \log (2\pi)^{n/2} - \frac{1}{2}\log |\Sigma_1|-\frac{1}{2}(\bold{x} - \boldsymbol{\mu}_1)^T \Sigma_1^{-1} (\bold{x} - \boldsymbol{\mu}_1)\\ &\quad + \log (2\pi)^{n/2} + \frac{1}{2}\log |\Sigma_2|+\frac{1}{2}(\bold{x} - \boldsymbol{\mu}_2)^T \Sigma_2^{-1} (\bold{x} - \boldsymbol{\mu}_2) \Big]\\ &= \frac{1}{2} \Bigg(\log \frac{|\Sigma_2|}{|\Sigma_1|}-\mathbb{E}_{p(\bold{x})}\Big[(\bold{x} - \boldsymbol{\mu}_1)^T \Sigma_1^{-1} (\bold{x} - \boldsymbol{\mu}_1) \Big] \\ &\quad+\mathbb{E}_{p(\bold{x})} \Big[ (\bold{x} - \boldsymbol{\mu}_2)^T \Sigma_2^{-1} (\bold{x} - \boldsymbol{\mu}_2)\Big] \Bigg)\\ &= \frac{1}{2} \Bigg(\log \frac{|\Sigma_2|}{|\Sigma_1|} -\mathbb{E}_{p(\bold{x})} \Big[tr\Big((\bold{x} - \boldsymbol{\mu}_1)^T \Sigma_1^{-1} (\bold{x} - \boldsymbol{\mu}_1)\Big) \Big] \\ &\quad +\mathbb{E}_{p(\bold{x})} \Big[tr\Big((\bold{x} - \boldsymbol{\mu}_2)^T \Sigma_2^{-1} (\bold{x} - \boldsymbol{\mu}_2)\Big) \Big] \Bigg) \\ &= \frac{1}{2} \Bigg(\log \frac{|\Sigma_2|}{|\Sigma_1|} -\mathbb{E}_{p(\bold{x})} \Big[tr\Big(\Sigma_1^{-1} (\bold{x} - \boldsymbol{\mu}_1)(\bold{x} - \boldsymbol{\mu}_1)^T \Big) \Big] \\ &\quad +\mathbb{E}_{p(\bold{x})} \Big[tr \Big(\Sigma_2^{-1} (\bold{x} - \boldsymbol{\mu}_2)(\bold{x} - \boldsymbol{\mu}_2)^T \Big) \Big] \Bigg) \\ &= \frac{1}{2} \Bigg(\log \frac{|\Sigma_2|}{|\Sigma_1|} -tr \Big( \Sigma_1^{-1} \mathbb{E}_{p(\bold{x})}\Big[ (\bold{x} - \boldsymbol{\mu}_1)(\bold{x} - \boldsymbol{\mu}_1)^T \Big]\Big) \\ &\quad +tr \Big( \Sigma_2^{-1} \mathbb{E}_{p(\bold{x})}\Big[ (\bold{x} - \boldsymbol{\mu}_2)(\bold{x} - \boldsymbol{\mu}_2)^T \Big]\Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - tr(\Sigma_1^{-1} \Sigma_1) \\ &\quad + tr \Big( \Sigma_2^{-1} \mathbb{E}_{p(\bold{x})}\Big[ (\bold{x}\bold{x}^T - 2 \bold{x} \boldsymbol{\mu}_2^T + \boldsymbol{\mu}_2 \boldsymbol{\mu}_2^T) \Big]\Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n \\ &\quad + tr \Big( \Sigma_2^{-1} \mathbb{E}_{p(\bold{x})}\Big[ (\Sigma_1+2\bold{x}\boldsymbol{\mu}_1^T - \boldsymbol{\mu}_1 \boldsymbol{\mu}_1^T - 2 \bold{x} \boldsymbol{\mu}_2^T + \boldsymbol{\mu}_2 \boldsymbol{\mu}_2^T) \Big]\Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n \\ &\quad + tr \Big( \Sigma_2^{-1} (\Sigma_1 + 2 \boldsymbol{\mu}_1 \boldsymbol{\mu}_1^T - \boldsymbol{\mu}_1 \boldsymbol{\mu}_1^T - 2 \boldsymbol{\mu}_1 \boldsymbol{\mu}_2^T + \boldsymbol{\mu}_2 \boldsymbol{\mu}_2^T) \Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n \\ &\quad + tr (\Sigma_2^{-1} \Sigma_1) + tr \Big( \Sigma_2^{-1} ( \boldsymbol{\mu}_1 \boldsymbol{\mu}_1^T - 2 \boldsymbol{\mu}_1 \boldsymbol{\mu}_2^T + \boldsymbol{\mu}_2 \boldsymbol{\mu}_2^T) \Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n \\ &\quad + tr (\Sigma_2^{-1} \Sigma_1) + tr \Big( \Sigma_2^{-1} (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2) (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2)^T \Big) \Bigg) \\ &=\frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n \\ &\quad + tr (\Sigma_2^{-1} \Sigma_1) + tr \Big( (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2)^T \Sigma_2^{-1} (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2) \Big) \Bigg) \\ &= \frac{1}{2} \Bigg( \log \frac{|\Sigma_2|}{|\Sigma_1|} - n + tr (\Sigma_2^{-1} \Sigma_1) + (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2)^T \Sigma_2^{-1} (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2) \Bigg) \end{aligned}

From​