Bayes公式告诉我们\begin{align*} p(y|x) = \frac{p(x|y)p(y)}{p(x)} \end{align*}其中$p(y)$被称为先验分布，是我们对随机变量$y$的初始信念；$p(x|y)$被称作似然，它表示的是在给定$y$的条件下$x$的概率，这个可从观测数据中得到；$p(y|x)$则是后验分布，它表示的是在看到数据后，我们对随机变量$y$的新的信念。

　　一般来说，给定一个似然，后验与先验不会属于同一个分布族，但也存在一些特例，当先验与后验同属一个分布族时，该先验称为似然的共轭先验，常见的有Gamma-Poisson共轭先验、Beta-Binomial共轭先验。

　　设随机变量$y$服从参数为$n,\alpha$的Gamma分布，即\begin{align*} p(y) = \frac{\alpha^n y^{n-1} e^{- \alpha y}}{(n-1)!} \end{align*}同时在给定$y$的条件下，随机变量$x$服从参数为$y$的Poisson分布，即\begin{align*} p(x = k|y) = \frac{y^k e^{-y}}{k!} \end{align*}于是\begin{align*} p(y|x = k) \varpropto p(y) p(x = k|y) & = \frac{\alpha^n y^{n-1} e^{- \alpha y}}{(n-1)!} \frac{y^k e^{-y}}{k!} = \frac{\alpha^n y^{n + k -1} e^{- (\alpha+1) y}}{(n-1)! k!} \varpropto \frac{(\alpha+1)^{n+k} y^{n + k -1} e^{- (\alpha+1) y}}{(n + k -1)!} \end{align*}注意$p(y|x = k)$是一个分布，故必然有\begin{align*} p(y|x = k) = \frac{(\alpha+1)^{n+k} y^{n + k -1} e^{- (\alpha+1) y}}{(n + k -1)!} \end{align*}即后验分布服从参数为$n + k, \alpha + 1$的Gamma分布。

　　设随机变量$y$服从参数为$\alpha$和$\beta$的Beta分布，即\begin{align*} p(y) = \frac{y^{\alpha - 1} (1 - y)^{\beta - 1}}{Beta(\alpha, \beta)} = \frac{(\alpha+\beta-1)!}{(\alpha-1)!(\beta-1)!} y^{\alpha - 1} (1 - y)^{\beta - 1} \end{align*}同时在给定$y$的条件下，随机变量$x$服从参数为$n$和$y$的Binomial分布，即\begin{align*} p(x = k|y) = \textrm{C}_n^k y^k (1-y)^{n-k} = \frac{n!}{k!(n-k)!} y^k (1-y)^{n-k} \end{align*}于是\begin{align*} p(y|x) \varpropto p(y) p(x = k|y) & = \frac{(\alpha+\beta-1)!}{(\alpha-1)!(\beta-1)!} y^{\alpha - 1} (1 - y)^{\beta - 1} \frac{n!}{k!(n-k)!} y^k (1-y)^{n-k} \\ & = \frac{(\alpha+\beta-1)!n!}{(\alpha-1)!(\beta-1)!k!(n-k)!} y^{\alpha + k - 1} (1 - y)^{\beta + n - k - 1} \\ & \varpropto \frac{y^{\alpha + k - 1} (1 - y)^{\beta + n - k - 1}}{Beta(\alpha + k, \beta + n - k)} \end{align*}注意$p(y|x)$是一个分布，故必然有\begin{align*} p(y|x) = \frac{y^{\alpha + k - 1} (1 - y)^{\beta + n - k - 1}}{Beta(\alpha + k, \beta + n - k)} \end{align*}即后验分布服从参数为$\alpha + k$和$\beta + n - k$的Beta分布。

　　Dirichlet分布是Beta分布的多元扩展，Multinomial分布是Binomial分布的多元扩展，不难猜到Dirichlet分布应该是Multinomial分布的共轭先验。设随机变量$\boldsymbol{y} = [y_1, \cdots, y_{k+1}]$服从参数为$\boldsymbol{\alpha} = [\alpha_1, \cdots, \alpha_{k+1}]$的Dirichlet分布，即\begin{align*} p(y_1, \cdots, y_{k+1}) = \frac{\Gamma(\alpha_{k+1} + \cdots + \alpha_1)}{\Gamma(\alpha_{k+1}) \cdots \Gamma(\alpha_1)} \prod_{i=1}^{k+1} y_i^{\alpha_i - 1} \end{align*}同时在给定$y$的条件下，随机变量$\boldsymbol{x} = [x_1, \cdots, x_{k+1}]$服从参数为$\boldsymbol{y} = [y_1, \cdots, y_{k+1}]$的Multinomial分布，即\begin{align*} p(x_1 = n_1, \cdots, x_{k+1} = n_{k+1} | y_1, \cdots, y_{k+1}) = \frac{\Gamma(n_{k+1} + \cdots + n_1)}{\Gamma(n_{k+1}) \cdots \Gamma(n_1)} \prod_{i=1}^{k+1} y_i^{n_i} \end{align*}记$\boldsymbol{n} = [n_1, \cdots, n_{k+1}]$，于是\begin{align*} p(\boldsymbol{y}|\boldsymbol{x} = \boldsymbol{n}) \varpropto p(\boldsymbol{y}) p(\boldsymbol{x} = \boldsymbol{n}|\boldsymbol{y}) & = \left( \frac{\Gamma(\alpha_{k+1} + \cdots + \alpha_1)}{\Gamma(\alpha_{k+1}) \cdots \Gamma(\alpha_1)} \prod_{i=1}^{k+1} y_i^{\alpha_i - 1} \right) \left( \frac{\Gamma(n_{k+1} + \cdots + n_1)}{\Gamma(n_{k+1}) \cdots \Gamma(n_1)} \prod_{i=1}^{k+1} y_i^{n_i} \right) \\ & = \frac{\Gamma(\alpha_{k+1} + \cdots + \alpha_1) \Gamma(n_{k+1} + \cdots + n_1)}{\Gamma(\alpha_{k+1}) \cdots \Gamma(\alpha_1) \Gamma(n_{k+1}) \cdots \Gamma(n_1)} \prod_{i=1}^{k+1} y_i^{\alpha_i + n_i - 1} \\ & \varpropto \frac{\Gamma(\alpha_{k+1} + n_{k+1} + \cdots + \alpha_1 + n_1)}{\Gamma(\alpha_{k+1} + n_{k+1}) \cdots \Gamma(\alpha_1 + n_1)} \prod_{i=1}^{k+1} y_i^{\alpha_i + n_i - 1} \end{align*}注意$p(\boldsymbol{y}|\boldsymbol{x} = \boldsymbol{n})$是一个分布，故必然有\begin{align*} p(\boldsymbol{y}|\boldsymbol{x} = \boldsymbol{n}) = \frac{\Gamma(\alpha_{k+1} + n_{k+1} + \cdots + \alpha_1 + n_1)}{\Gamma(\alpha_{k+1} + n_{k+1}) \cdots \Gamma(\alpha_1 + n_1)} \prod_{i=1}^{k+1} y_i^{\alpha_i + n_i - 1} \end{align*}即后验分布服从参数为$\boldsymbol{\alpha} + \boldsymbol{n}$的Dirichlet分布。

共轭先验