1. Ensemble classifier 

Each classifier votes on prediction

Ensemble model = sign(w₁f₁(x_i) + w₂f₂(x_i) + w₃f₃(x_i))

w₁ w₂ w_{3 is the learning coefficients}

f₁(x_i), f₂(x_i), f₃(x_i)) is three classifiers

2. Boosting

Focus on hard or more important pointsand keep adding new classfier.

Boosting is more robust to overfitting but we still need carefully to choose boosting captical T

using validation set or cross validation.

3. Adaboost

1. Start with weight for all points: α_i = 1/N

For t = 1 ... T

• Learn f_t(x) with data weights α_i
• Compute coefficient w_t 
  • Note :

    Adaboost use the formual below to compute coefficient w_t of classifier f_t(x)

    w_t  = 1/2*ln(1- weighted_error(f_t)/weighted_error(f_t))

• Recompute weights α_i
  • 　　α_i = α_ie^-W_t, if f_t(x_i)=y_i else α_ie^W_t
• Normalizing weights:
  • 　　α_i = αi / (α₁ +α₂ ... 　α_N)

Final model predicts the value by:

y = sign(w₁ f₁(x) + w_t f_t(x) ... w_T f_T(x))

Weighted classification error:

weighted_error = total weight of mistakes / total weights of all data points

Normalizing weights α_i

normalize weights to add up to 1 after every iterationn

α_i = αi / (α₁ +α₂ ... 　α_N)

4. Adaboost Theorem

if we can find a weak leatner with weighted_error < 0.5 (beat random guess) at every iteration t,

 the training error of boosted classifier goes to zero as the iterations of boosting goes to infinity.

机器学习笔记（Washington University）- Classification Specialization-week five