TABLE OF CONTENTS

EQUATION LIST

nlp

• Conditional Random Field CRF

  #machine learning #nlp
  $$P(y|x)=\frac{1}{Z(x)}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i)) \\ Z(x)=\sum_{y}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i))$$
  READ MORE

• Hidden Markov Model

  #machine learning #nlp
  $$Q=\{q_{1},q_{2},...,q_{N}\}, V=\{v_{1},v_{2},...,v_{M}\} \\ I=\{i_{1},i_{2},...,i_{T}\},O=\{o_{1},o_{2},...,o_{T}\} \\ A=[a_{ij}]_{N \times N}, a_{ij}=P(i_{t+1}=q_{j}|i_{t}=q_{i}) \\ B=[b_{j}(k)]_{N \times M},b_{j}(k)=P(o_{t}=v_{k}|i_{t}=q_{j})$$
  READ MORE

• Transformer

  #machine learning #nlp #gpt
  $$\text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$$
  READ MORE

machine learning

• Area Under Uplift Curve AUUC

  #machine learning #causual inference
  $$f(t)=(\frac{Y^{T}_{t}}{N^{T}_{t}} - \frac{Y^{C}_{t}}{N^{C}_{t}})(N^{T}_{t}+N^{C}_{t})$$
  READ MORE

• Average Treatment Effect ATE

  #machine learning #causual inference
  $$\text{ATE}:=\mathbb{E}[Y(1)-Y(0)]$$
  READ MORE

• Bellman Equation

  #machine learning
  $$v_{\pi}(s)=\sum_{a}\pi(a|s)\sum_{s^{'},r}p(s^{'},r|s,a)[r+\gamma v_{\pi}(s^{'})]$$
  READ MORE

• Conditional Average Treatment Effect CATE

  #machine learning #causual inference
  $$\tau(x):=\mathbb{E}[Y(1)-Y(0)|X=x]$$
  READ MORE

• Deep Kernel Learning

  #machine learning #Deep Kernel Learning
  $$k(x_{i},x_{j}|\phi)=k(h(x_i,w_k),h(x_j,w_k)|w_k,\phi)$$
  READ MORE

• Diffusion Model Forward Process

  #machine learning #diffusion
  $$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I) \\q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})$$
  READ MORE

• Diffusion Model Forward Process Reparameterization

  #machine learning #diffusion
  $$x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}} \epsilon_{t-1}\\=\sqrt{\alpha_{t}\alpha_{t-1}}x_{t-2} + \sqrt{1-\alpha_{t}\alpha_{t-1}} \bar{\epsilon}_{t-2}\\=\text{...}\\=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon \\\alpha_{t}=1-\beta_{t}, \bar{\alpha}_{t}=\prod_{t=1}^{T}\alpha_{t}$$
  READ MORE

• Diffusion Model Reverse Process

  #machine learning #diffusion
  $$p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod^T_{t=1} p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) \\ p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))$$
  READ MORE

• Diffusion Model Variational Lower Bound

  #machine learning #diffusion
  $$\begin{aligned} - \log p_\theta(\mathbf{x}_0) &\leq - \log p_\theta(\mathbf{x}_0) + D_\text{KL}(q(\mathbf{x}_{1:T}\vert\mathbf{x}_0) \| p_\theta(\mathbf{x}_{1:T}\vert\mathbf{x}_0) ) \\ &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_{\mathbf{x}_{1:T}\sim q(\mathbf{x}_{1:T} \vert \mathbf{x}_0)} \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T}) / p_\theta(\mathbf{x}_0)} \Big] \\ &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_q \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} + \log p_\theta(\mathbf{x}_0) \Big] \\ &= \mathbb{E}_q \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \\ \text{Let }L_\text{VLB} &= \mathbb{E}_{q(\mathbf{x}_{0:T})} \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \geq - \mathbb{E}_{q(\mathbf{x}_0)} \log p_\theta(\mathbf{x}_0) \end{aligned}$$
  READ MORE

• Diffusion Model Variational Lower Bound Loss

  #machine learning #diffusion
  $$\begin{aligned} L_\text{VLB} &= L_T + L_{T-1} + \dots + L_0 \\ \text{where } L_T &= D_\text{KL}(q(\mathbf{x}_T \vert \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_T)) \\ L_t &= D_\text{KL}(q(\mathbf{x}_t \vert \mathbf{x}_{t+1}, \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_t \vert\mathbf{x}_{t+1})) \text{ for }1 \leq t \leq T-1 \\ L_0 &= - \log p_\theta(\mathbf{x}_0 \vert \mathbf{x}_1) \end{aligned}$$
  READ MORE

• Generative Adversarial Networks GAN

  #machine learning #gan
  $$\min_{G} \max_{D} V(D,G)=\mathbb{E}_{x \sim p_{data}(x)}[\log D(x)]+\mathbb{E}_{z \sim p_{z}(z)}[\log(1-D(G(z)))]$$
  READ MORE

• Individual Treatment Effect ITE

  #machine learning #causual inference
  $$\text{ITE}_{i}:=Y_{i}(1)-Y_{i}(0)$$
  READ MORE

• Jensen-Shannon Divergence JS-Divergence

  #machine learning
  $$JS(P||Q)=\frac{1}{2}KL(P||\frac{(P+Q)}{2})+\frac{1}{2}KL(Q||\frac{(P+Q)}{2})$$
  READ MORE

• KL-Divergence

  #machine learning
  $$KL(P||Q)=\sum_{x}P(x)\log(\frac{P(x)}{Q(x)})$$
  READ MORE

• Kullback-Leibler Divergence

  #machine learning #kl divergence
  $$KL(P||Q)=\sum_{x}P(x)\log(\frac{P(x)}{Q(x)})$$
  READ MORE

• Mahalanobis Distance

  #machine learning #mahalanobis
  $$D_{M}(x,y)=\sqrt{(x-y)^{T}\Sigma^{-1}(x-y)}$$
  READ MORE

• Maximum Mean Discrepancy MMD

  #machine learning #mmd
  $$\textup{MMD}(\mathbb{F},X,Y):=\sup_{f \in\mathbb{F}}(\frac{1}{m}\sum_{i=1}^{m}f(x_{i}) -\frac{1}{n}\sum_{j=1}^{n}f(y_{j}))$$
  READ MORE

• Propensity Score

  #machine learning #causual inference
  $$e := p(W=1|X=x)$$
  READ MORE

• QINI

  #machine learning #causual inference
  $$g(t)=Y^{T}_{t}-\frac{Y^{C}_{t}N^{T}_{t}}{N^{C}_{t}},\\ f(t)=g(t) \times \frac{N^{T}_{t}+N^{C}_{t}}{N^{T}_{t}}$$
  READ MORE

• S-Learner

  #machine learning #causual inference
  $$\mu(x,w)=\mathbb{E}[Y_{i}|X=x_{i},W=w] \\ \hat{\tau}(x)=\hat{\mu}(x,1)-\hat{\mu}(x,0)$$
  READ MORE

• Support Vector Machine SVM

  #machine learning #svm
  $$\max_{w,b} \frac{2}{||w||} \\ s.t.\ y_{i}(w^{T}x_{i} + b) \geq 1, i=1,2,...,m \\ L(w,b,\alpha)=\frac{1}{2}||w||^2 + \sum^{m}_{i=1}a_{i}(1-y_{i}(w^{T}x_{i} + b))$$
  READ MORE

• T-Learner

  #machine learning #causual inference
  $$\mu_{0}(x)=\mathbb{E}[Y(0)|X=x],\mu_{1}(x)=\mathbb{E}[Y(1)|X=x],\\ \hat{\tau}(x)=\hat{\mu}_{1}(x)-\hat{\mu}_{0}(x)$$
  READ MORE

• TransE

  #machine learning
  $$\mathcal{L}=\sum_{(h,r,t) \in S} \sum_{(h^{'},r^{'},t^{'}) \in S^{'}_{(h,r,t)}} \[ \gamma + d(h + r, t) - d(h^{'} + r^{'}, t^{'}) \]_{+} \\ S^{'}_{(h,r,t)}=\{(h^{'},r,t)|h^{'} \in E\} \cup \{(h,r,t^{'})|t^{'} \in E\} \\ d(h + r, t)=||h + r - t||^{2}_{2}$$
  READ MORE

• TransH

  #machine learning #KG
  $$f_{r}(h,t) =||h_{\perp} + d_{r} - t_{\perp} ||^{2}_{2}=||(h - w_{r}hw_{r}) + d_{r} - (t - w_{r}tw_{r}) ||^{2}_{2}$$
  READ MORE

• Unconfoundedness Assumption

  #machine learning #causual inference
  $$\{Y_{i}(0),Y_{i}(1)\}\perp W_{i}|X_{i}$$
  READ MORE

• Variational AutoEncoder VAE

  #machine learning #VAE
  $$\log p_{\theta}(x)=\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x)] \\ =\mathbb{E}_{q_{\phi}(z|x)}[\log \frac{p_{\theta}(x,z)}{p_{\theta}(z|x)}] \\ =\mathbb{E}_{q_{\phi}(z|x)}[\log [\frac{p_{\theta}(x,z)}{q_{\phi}(z|x)} \times \frac{q_{\phi}(z|x)}{p_{\theta}(z|x)}]] \\ =\mathbb{E}_{q_{\phi}(z|x)}[\log [\frac{p_{\theta}(x,z)}{q_{\phi}(z|x)} ]] +D_{KL}(q_{\phi}(z|x) || p_{\theta}(z|x))\\$$
  READ MORE

• Wasserstein Distance Optimal Transport

  #machine learning #wasserstein
  $$W_{p}(P,Q)=(\inf_{J \in J(P,Q)} \int{||x-y||^{p}dJ(X,Y)})^\frac{1}{p}$$
  READ MORE

• X-Learner

  #machine learning #causual inference
  $$\tilde{D}^{1}_{i}:=Y^{1}_{i}-\hat{\mu}_{0}(X^{1}_{i}),\tilde{D}^{0}_{i}:=\hat{\mu}_{1}(X^{0}_{i})-Y^{0}_{i}\\ \hat{\tau}(x)=g(x)\hat{\tau}_{0}(x) + (1-g(x))\hat{\tau}_{1}(x)$$
  READ MORE

• Bound on Target Domain Error

  #machine learning #transfer learning
  $$\epsilon_{T}(h) \le \hat{\epsilon}_{S}(h) + \sqrt{\frac{4}{m}(d \log \frac{2em}{d} + \log \frac{4}{\delta })} + d_{\mathcal{H}}(\tilde{\mathcal{D}}_{S}, \tilde{\mathcal{D}}_{T}) + \lambda \\ \lambda = \lambda_{S} + \lambda_{T}$$
  READ MORE

• Bregman Divergences

  #machine learning
  $$d_{\phi}(z,z^{'})=\phi(z) - \phi(z^{'})-(z-z^{'})^{T} \nabla \phi(z^{'})$$
  READ MORE

• Coefficient of Determination

  #machine learning #metric
  $$SS_{res} = \sum_{i} (y_{i} - f_{i})^{2} = \sum_{i} e_{i}^{2}, SS_{total} = \sum_{i} (y_{i} - \bar{y} )^{2}, y=\frac{1}{n} \sum^{n}_{i=1} y_{i}, R^{2} = 1 - \frac{SS_{res}}{SS_{total}}$$
  READ MORE

• Conditional Random Field CRF

  #machine learning #nlp
  $$P(y|x)=\frac{1}{Z(x)}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i)) \\ Z(x)=\sum_{y}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i))$$
  READ MORE

• Cross-Stitch Network

  #machine learning #multi task
  $$\begin{bmatrix} \tilde{x}^{ij}_{A}\\\tilde{x}^{ij}_{B}\end{bmatrix}=\begin{bmatrix} a_{AA} & a_{AB}\\ a_{BA} & a_{BB} \end{bmatrix}\begin{bmatrix} x^{ij}_{A}\\ x^{ij}_{B} \end{bmatrix}$$
  READ MORE

• Domain Adaptation H-Divergence

  #machine learning #transfer learning
  $$d_{\mathcal{H}}(\mathcal{D},\mathcal{D}^{'})=2\sup_{h \in \mathcal{H}}|\Pr_{\mathcal{D}}[I(h)]-\Pr_{\mathcal{D}^{'}}[I(h)]|$$
  READ MORE

• Domain-Adversarial Neural Networks DANN

  #machine learning #transfer learning
  $$\min [\frac{1}{m}\sum^{m}_{1}\mathcal{L}(f(\textbf{x}^{s}_{i}),y_{i})+\lambda \max(-\frac{1}{m}\sum^{m}_{i=1}\mathcal{L}^{d}(o(\textbf{x}^{s}_{i}),1)-\frac{1}{m^{'}}\sum^{m^{'}}_{i=1}\mathcal{L}^{d}(o(\textbf{x}^{t}_{i}),0))]$$
  READ MORE

• Entire Space Multi-Task Model ESSM

  #machine learning #multi task
  $$L(\theta_{cvr},\theta_{ctr})=\sum^{N}_{i=1}l(y_{i},f(x_{i};\theta_{ctr}))+\sum^{N}_{i=1}l(y_{i}\&z_{i},f(x_{i};\theta_{ctr}) \times f(x_{i};\theta_{cvr})) $$
  READ MORE

• Graph Attention Network GAT

  #machine learning #graph #GNN
  $$h=\{\vec{h_{1}},\vec{h_{2}},...,\vec{h_{N}}\}, \\ \vec{h_{i}} \in \mathbb{R}^{F} \\ W \in \mathbb{R}^{F \times F^{'}} \\ e_{ij}=a(Wh_{i},Wh_{j}) \\ k \in \mathcal{N}_{i},\text{ neighbourhood nodes}\\ a_{ij}=\text{softmax}_{j}(e_{ij})=\frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_{i}} \exp(e_{ik})}$$
  READ MORE

• Graph Convolutional Networks GCN

  #machine learning #graph #GNN
  $$H^{(l+1)}=\sigma(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}H^{l}W^{l})\\ \tilde{A}=A+I_{N}\\ \tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij} \\ H^{0}=X \\ \mathcal{L}=-\sum_{l \in Y}\sum^{F}_{f=1} Y_{lf} \ln Z_{lf}$$
  READ MORE

• Graph Laplacian

  #machine learning #graph #GNN
  $$L=I_{N}-D^{-\frac{1}{2}}AD^{-\frac{1}{2}} \\ L=U\Lambda U^{T}$$
  READ MORE

• GraphSage

  #machine learning #graph #GNN
  $$h^{0}_{v} \leftarrow x_{v} \\ \textbf{for} k \in \{1,2,...,K\} \text{do}\\ \textbf{for} v \in V \text{do} \\ h^{k}_{N_{v}} \leftarrow \textbf{AGGREGATE}_{k}(h^{k-1}_{u}, u \in N(v)); \\ h^{k}_{v} \leftarrow \sigma (W^{k} \textbf{concat}(h^{k-1}_{v},h^{k}_{N_{v}})) \\ \textbf{end} \\ h^{k}_{v}=h^{k}_{v}/||h^{k}_{v}||_{2},\forall v \in V \\ \textbf{end} \\ z_{v} \leftarrow h^{k}_{v} \\ J_{\textbf{z}_{u}}=-\log (\sigma (\textbf{z}_{u}^{T}\textbf{z}_{v})) - Q \mathbb{E}_{v_{n} \sim p_n(v)} \log(\sigma (-\textbf{z}_{u}^{T}\textbf{z}_{v_{n}}))$$
  READ MORE

• Hidden Markov Model

  #machine learning #nlp
  $$Q=\{q_{1},q_{2},...,q_{N}\}, V=\{v_{1},v_{2},...,v_{M}\} \\ I=\{i_{1},i_{2},...,i_{T}\},O=\{o_{1},o_{2},...,o_{T}\} \\ A=[a_{ij}]_{N \times N}, a_{ij}=P(i_{t+1}=q_{j}|i_{t}=q_{i}) \\ B=[b_{j}(k)]_{N \times M},b_{j}(k)=P(o_{t}=v_{k}|i_{t}=q_{j})$$
  READ MORE

• Huber Loss

  #machine learning
  $$L_{\delta}(y,f(x)) = \left \{ \begin{aligned} & \frac{1}{2}(y-f(x))^{2}, \text{for} |y-f(x)| \le \delta \cr & \delta \times (|y-f(x)| - \frac{1}{2}\delta) \cr \end{aligned} \right. $$
  READ MORE

• Language Modelling

  #machine learning
  $$p(x)=\prod^{n}_{i=1} p(s_{n}|s_{1},...,s_{n-1})$$
  READ MORE

• Mean Absolute Error MAE

  #machine learning #metric
  $$\text{MAE} = \frac{1}{n} \sum^{n}_{i=1} |Y_{i} - \hat{Y}_{i}| = \frac{1}{n} \sum^{n}_{i=1} |e_{i}|$$
  READ MORE

• Mean Squared Error MSE

  #machine learning #metric
  $$\text{MSE} = \frac{1}{n} \sum^{n}_{i=1} (Y_{i} - \hat{Y}_{i}) ^ {2}$$
  READ MORE

• Model-Agnostic Meta-Learning MAML

  #machine learning #meta learning
  $$\min_{\theta} \sum_{\mathcal{T}_{i} \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_{i}}(f_{\theta^{'}_{i}}) = \sum_{\mathcal{T}_{i} \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_{i}}(f_{\theta_{i} - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}_{i}} (f_{\theta}) })$$
  READ MORE

• Multi-Gate Mixture of Experts MMoE

  #machine learning #multi task
  $$g^{k}(x)=\text{softmax}(W_{gk}x) \\ f^{k}(x)=\sum^{n}_{i=1}g^{k}(x)_{i}f_{i}(x) \\ y_{k}=h^{k}(f^{k}(x))$$
  READ MORE

• Progressive Layered Extraction PLE

  #machine learning #multi task
  $$g^{k}(x)=w^{k}(x)S^{k}(x) \\ w^{k}(x)=\text{softmax}(W^{k}_{g}x) \\ S^{k}(x)=\[E^{T}_{(k,1)},E^{T}_{(k,2)},...,E^{T}_{(k,m_{k})},E^{T}_{(s,1)},E^{T}_{(s,2)},...,E^{T}_{(s,m_{s})}\]^{T} \\ y^{k}(x)=t^{k}(g^{k}(x)) \\ g^{k,j}(x)=w^{k,j}(g^{k,j-1}(x))S^{k,j}(x) $$
  READ MORE

• Prototypical Networks Protonets

  #machine learning #meta learning
  $$c_{k}=\frac{1}{|S_{k}|}\sum_{(x_{i},y_{i}) \in S_{k}} f_{\phi}(x) \\ p_{\phi}(y=k|x)=\frac{\exp(-d(f_{\phi}(x), c_{k}))}{\sum_{k^{'}} \exp(-d(f_{\phi}(x), c_{k^{'}})} \\\min J(\phi)=-\log p_{\phi}(y=k|x)$$
  READ MORE

• Proximal Policy Optimization

  #machine learning #AI #LLM
  $$\arg\max\limits_{\pi}{ E_{p \sim D,g \sim \pi} [R(g|p)] }, R(g|p) = \tilde{R}_{c}(g|p) - \beta D_{KL}( \pi_{\theta} (g|p) || \pi_{0} (g|p))$$
  READ MORE

• Proximal Policy Optimization PPO

  #machine learning
  $$L^{CLIP}(\theta)=E_{t}[\min(r_{t}(\theta))A_{t}, \text{clip}(r_{t}(\theta), 1-\epsilon,1+\epsilon)A_{t}]$$
  READ MORE

• RotatE

  #machine learning #KG
  $$f_{r}(h, t) = ||h \circ r - t||^{2}_{2}$$
  READ MORE

• SME Linear

  #machine learning #KG
  $$\epsilon(lhs,rel,rhs)=E_{lhs(rel)}^{T}E_{rhs(rel)} \\=(W_{l1}E_{lhs}^{T} + W_{l2}E_{rel}^{T} + b_{l})^{T}(W_{r1}E_{rhs}^{T} + W_{r2}E_{rel}^{T} + b_{r})$$
  READ MORE

• SimplE

  #machine learning #KG
  $$s(e_{i}, r, e_{j}) = \frac{1}{2}( + )$$
  READ MORE

• TransR

  #machine learning #KG
  $$h_{r}=hM_{r}, t_{r}=tM_{r} \\f_{r}(h, t) = ||h_{r} + r - t_{r}||^{2}_{2}=||hM_{r}+r-tM_{r}||^{2}_{2}$$
  READ MORE

• Transformer

  #machine learning #nlp #gpt
  $$\text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$$
  READ MORE