Latex Code for Diffusion Models Equations

rockingdingo 2022-09-18 #Diffusion #VAE #GAN #Generative Models
0
0


Latex Code for Diffusion Models Equations

Navigation

In this blog, we will summarize the latex code of equations for Diffusion Models, which are among the top-performing generative models, including GAN, VAE and flow-based models. The basic idea of diffusion models are to inject random noise to the feature vector in the forward process as markov chain models, and in the reverse process gradualy reconstruct the feature vector for generation. See below blogpost as reference for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. lilianweng.github.io/posts/2021-07-11-diffusion-models/.

1. Diffusion Model

  • 1.1 Forward Process

    Equation


    Latex Code
                q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I) \\q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})
    Explanation

    See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

  • 1.2 Forward Process Reparameterization Trick

    Equation


    Latex Code
                x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}} \epsilon_{t-1}\\=\sqrt{\alpha_{t}\alpha_{t-1}}x_{t-2} + \sqrt{1-\alpha_{t}\alpha_{t-1}} \bar{\epsilon}_{t-2}\\=\text{...}\\=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon \\\alpha_{t}=1-\beta_{t}, \bar{\alpha}_{t}=\prod_{t=1}^{T}\alpha_{t}
    Explanation

    See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

  • 1.3 Reverse Process

    Equation


    Latex Code
                p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod^T_{t=1} p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) \\
            p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))
    Explanation

    See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

  • 1.4 Reverse Process Variational Lower Bound

    Equation


    Latex Code
                \begin{aligned}
            - \log p_\theta(\mathbf{x}_0) 
            &\leq - \log p_\theta(\mathbf{x}_0) + D_\text{KL}(q(\mathbf{x}_{1:T}\vert\mathbf{x}_0) \| p_\theta(\mathbf{x}_{1:T}\vert\mathbf{x}_0) ) \\
            &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_{\mathbf{x}_{1:T}\sim q(\mathbf{x}_{1:T} \vert \mathbf{x}_0)} \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T}) / p_\theta(\mathbf{x}_0)} \Big] \\
            &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_q \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} + \log p_\theta(\mathbf{x}_0) \Big] \\
            &= \mathbb{E}_q \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \\
            \text{Let }L_\text{VLB} 
            &= \mathbb{E}_{q(\mathbf{x}_{0:T})} \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \geq - \mathbb{E}_{q(\mathbf{x}_0)} \log p_\theta(\mathbf{x}_0)
            \end{aligned}
    Explanation

    See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

  • 1.5 Reverse Process Variational Lower Bound Loss Function

    Equation


    Latex Code
                \begin{aligned}
            L_\text{VLB} &= L_T + L_{T-1} + \dots + L_0 \\
            \text{where } L_T &= D_\text{KL}(q(\mathbf{x}_T \vert \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_T)) \\
            L_t &= D_\text{KL}(q(\mathbf{x}_t \vert \mathbf{x}_{t+1}, \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_t \vert\mathbf{x}_{t+1})) \text{ for }1 \leq t \leq T-1 \\
            L_0 &= - \log p_\theta(\mathbf{x}_0 \vert \mathbf{x}_1)
            \end{aligned}
    Explanation

    See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.




Related Contents