Loss Function

Loss function is used to measure the degree of fit. So for machine learning a few elements are:

1. Hypothesis space: e.g. parametric form of the function such as linear regression, logistic regression, svm, etc.
2. Measure of fit: loss function, likelihood
3. Tradeoff between bias vs. variance: regularization. Or bayesian estimator (MAP)
4. Find a good h in hypothesis space: optimization. convex - global. non-convex - multiple starts
5. Verification of h: predict on test data. cross validation.

Among all linear methods y=f(θTx)y=f(θTx), we need to first determine the form of ff, and then finding θθ by formulating it to maximizing likelihood or minimizing loss. This is straightforward.

For classification, it‘s easy to see that if we classify correctly we have y⋅f=y⋅θTx>0y⋅f=y⋅θTx>0, and y⋅f=y⋅θTx<0y⋅f=y⋅θTx<0 if incorrectly. Then we formulate following loss functions:

1. 0/1 loss: minθ∑iL0/1(θTx)minθ∑iL0/1(θTx). We define L0/1(θTx)=1L0/1(θTx)=1 if y⋅f<0y⋅f<0, and =0=0 o.w. Non convex and very hard to optimize.
2. Hinge loss: approximate 0/1 loss by minθ∑iH(θTx)minθ∑iH(θTx). We define H(θTx)=max(0,1−y⋅f)H(θTx)=max(0,1−y⋅f). Apparently HH is small if we classify correctly.
3. Logistic loss: minθ∑ilog(1+exp(−y⋅θTx))minθ∑ilog(1+exp?(−y⋅θTx)). Refer to my logistic regression notes for details.

For regression:

1. Square loss: minθ∑i||y(i)−θTx(i)||2minθ∑i||y(i)−θTx(i)||2

Fortunately, hinge loss, logistic loss and square loss are all convex functions. Convexity ensures global minimum and it‘s computationally appleaing.

https://www.kaggle.com/wiki/LogarithmicLoss

损失函数