1. Probabilistic clustering model

• (k-means) Hard assignments do not tell the full story, capture the uncertainty
• k-means only considers the cluster center, not good for overlapping clusters,disparate cluster size,different shaped cluster
• learn weights on dimensions
• can learn cluster-specific weights on dimensions

2. Gaussian distribution

1-D gaussian is fully specified by mean μ and variance σ².

2-D gaussian is fully specified by mean μ vector and covariance matrix Σ.

 thusly our mixture model of gaussian is defined by 

{π_k, μ_k, Σ_k} 

3. EM(Expectation maximization)

what if we knew the cluster parameters {π_k, μ_k, Σ_k} ?

compute responsibilites:

r_ik is the responsibility cluster k takes for observation i.

p is the probability of assignment to cluster k, given model parameters and observaed value.

π_k is the initial probability of being from cluster k.

N is the gaussian model.

what if we knew the cluster soft assignments r_ij ?

The procedure for the iterative algorithm:

1. initialize

2. estimate cluster responsibilities given current parameter estimates(E-step)

3. maximize likelihood given soft assignments

Notes:

EM is a coordinate-ascent algorithm

EM converges to a local mode

There are many ways to initialize the EM algorithm and it is important for convergence rates and quality of local mode

• random choose k centroids
• pick center sequentially like in k-means++
• initilize k-means solution
• grow mixture model by splitting until k clusters are formed

prevent overfitting

• Do not let the variance goes down to zero, add small amount to diagonal of covariance estimate

机器学习笔记（Washington University）- Clustering Specialization-week four