首页 > 代码库 > Deep Learning Tutorial - Classifying MNIST digits using Logistic Regression

Deep Learning Tutorial - Classifying MNIST digits using Logistic Regression

      Deep Learning Tutorial 由 Montreal大学的LISA实验室所作,基于Theano的深度学习材料.Theano是一个python库,使得写深度模型更容易些,也可以在GPU上训练深度模型。所以首先得了解python和numpy。其次,阅读Theano basic tutorial。

     Deep Learning Tutorial 包括:

       监督学习算法:

            Logistic Regression - using Theano for something simple

            Multilayer perceptron - introduction to layers

            Deep Convolutional Network - a simplified version of LeNet5

       无监督和半监督学习算法:

            Auto Encoders, Denoising Autoencoders - description of autoencoders

            Stacked Denoising Auto-Encoders - easy steps into unsupervised pre-training for deep nets

            Restricted Boltzmann Machines - single layer generative RBM model

            Deep Belief Networks - unsupervised generative pre-training of stacked RBMs followed by supervised fine-tuning
     
  建立mcRBM模型:
           
 HMC Sampling

        其他:

              Contractive auto-encoders:

              Semantic Parsing of Speech using Recurrent Net

              LSTM network

              Modeling and generating sequences of polyphonic music

 

 

1.Getting started:

      存储数据时用shared variables(共享变量),用小批量索引来操作数据。共享变量与利用GPU有关,如果数据时共享变量,Theano可以复制整个数据在GPU上通过一个简单的命令,不需要从CPU存储上复制任何信息。当在GPU上存储数据时需要的类型是floats,因此代码type为floatX。由于y为标签,所以要将y cast为int。

 

 2.损失函数:  

     0-1损失:  技术分享

     

    对数似然损失:技术分享,因为0-1损失函数不可微,我们一般用最大化对数似然损失,即最小化负的对     数似然损失。

 

3.梯度下降

    批梯度、随机梯度、小批量梯度

4.正则化

    L1,L2正则化(权重衰减)

5.提前停止(Early-Stopping

     之前没注意这个算法,后面的logistic回归分类手写数字识别时用到了它。思想大致是:在验证数据集上如果连续多次迭代过程中损失函数不再显著降低,那么应该提前结束训练,同时也可防止过拟合。

6.Theano tips

      若要存储在程序运行过程后所优化得到的权重参数,利用pickle或深拷贝,如果参数为共享变量w,v,u,可这样操作:

>>> import cPickle
>>> save_file = open(’path’, ’wb’) # this will overwrite current contents
>>> cPickle.dump(w.get_value(borrow=True), save_file, -1) # the -1 is for HIGHEST_PROTOCOL
>>> cPickle.dump(v.get_value(borrow=True), save_file, -1) # .. and it triggers much more efficient
>>> cPickle.dump(u.get_value(borrow=True), save_file, -1) # .. storage than numpy’s default
>>> save_file.close()

     若要用这些w,v,u来预测或评价,可以再load一下:

>>> save_file = open(’path’)   
>>> w.set_value(cPickle.load(save_file), borrow=True)
>>> v.set_value(cPickle.load(save_file), borrow=True)
>>> u.set_value(cPickle.load(save_file), borrow=True)

     建议不要pickle训练或测试函数为长期储存,Theano函数与python深拷贝和pickle机制兼容,如果你更新了Theano文件使其内部改变,那么可能将无法在un-pickle你的模型。

  

      CLASSIFYING MNIST DIGITS USING LOGISTIC REGRESSION

 logistic regression模型(打分or评价函数):

 技术分享

  logistic regression模型(损失函数):

技术分享

整体代码:

  

#coding=UTF-8
import cPickle
import gzip
import os
import sys
import timeit
import numpy
import theano
import theano.tensor as T


class LogisticRegression(object):   #此类有三个函数
    def __init__(self, input, n_in, n_out):        
        self.W = theano.shared(value=http://www.mamicode.com/numpy.zeros((n_in, n_out), dtype=theano.config.floatX),name=W,borrow=True)  #初始化参数
        self.b = theano.shared(value=http://www.mamicode.com/numpy.zeros((n_out,), dtype=theano.config.floatX), name=b, borrow=True)
        self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W) + self.b)   #计算打分、评价结果
        self.y_pred = T.argmax(self.p_y_given_x, axis=1)      #挑出可能性最大的类别标签
        self.params = [self.W, self.b]
        self.input = input

    def negative_log_likelihood(self, y):     
        return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])    #计算损失函数

    def errors(self, y):        #计算错误率
        if y.ndim != self.y_pred.ndim:
            raise TypeError(y should have the same shape as self.y_pred, (y, y.type, y_pred, self.y_pred.type))
        if y.dtype.startswith(int):
            return T.mean(T.neq(self.y_pred, y))
        else:
            raise NotImplementedError()


def load_data(dataset):    
    f = gzip.open(dataset,rb)
    train_set, valid_set, test_set = cPickle.load(f)
    f.close()    #下载数据
 
    def shared_dataset(data_xy, borrow=True):   #将数据类型改为共享数据类型
        data_x, data_y = data_xy
        shared_x = theano.shared(numpy.asarray(data_x, dtype=theano.config.floatX), borrow=borrow)
        shared_y = theano.shared(numpy.asarray(data_y, dtype=theano.config.floatX), borrow=borrow)
        return shared_x, T.cast(shared_y, int32)

    test_set_x, test_set_y = shared_dataset(test_set)
    valid_set_x, valid_set_y = shared_dataset(valid_set)    
    train_set_x, train_set_y = shared_dataset(train_set)
    rval = [(train_set_x, train_set_y), (valid_set_x, valid_set_y), (test_set_x, test_set_y)]   #改好的三种共享数据类型
    return rval

def sgd_optimization_mnist(learning_rate=0.13, n_epochs=1000, dataset=data/mnist.pkl.gz, batch_size=600):   #随机梯度下降
    datasets = load_data(dataset)                     #学习速率0.13,迭代次数1000,批梯度大小600
    train_set_x, train_set_y = datasets[0]
    valid_set_x, valid_set_y = datasets[1]
    test_set_x, test_set_y = datasets[2]
    n_train_batches = train_set_x.get_value(borrow=True).shape[0] / batch_size       
    n_valid_batches = valid_set_x.get_value(borrow=True).shape[0] / batch_size
    n_test_batches = test_set_x.get_value(borrow=True).shape[0] / batch_size        #计算三种数据的批量数目,若train_set有6000个样本,批梯度大小为600,
    print ...building the model‘                                                   #则n_train_batches为10.
    index = T.lscalar()
    x = T.matrix(x)
    y = T.ivector(y)
    classifier = LogisticRegression(input=x, n_in=28 * 28, n_out=10)    #Classifer为这个分类器
    cost = classifier.negative_log_likelihood(y)
    test_model = theano.function(     #测试模型
        inputs=[index],
        outputs=classifier.errors(y),
        givens={
            x: test_set_x[index * batch_size: (index + 1) * batch_size],
            y: test_set_y[index * batch_size: (index + 1) * batch_size]
        }
    )
    validate_model = theano.function(   #验证模型
        inputs=[index],
        outputs=classifier.errors(y),    #输出为错误率
        givens={
            x: valid_set_x[index * batch_size: (index + 1) * batch_size],
            y: valid_set_y[index * batch_size: (index + 1) * batch_size]
        }
    )
    g_W = T.grad(cost=cost, wrt=classifier.W)
    g_b = T.grad(cost=cost, wrt=classifier.b)
    updates = [(classifier.W, classifier.W - learning_rate * g_W), (classifier.b, classifier.b - learning_rate * g_b)]
    train_model = theano.function(    #训练模型
        inputs=[index],
        outputs=cost,
        updates=updates,
        givens={
            x: train_set_x[index * batch_size: (index + 1) * batch_size],
            y: train_set_y[index * batch_size: (index + 1) * batch_size]
        }
    )
    print ...training the model‘          #以下运用了early-stop算法,并非普通的迭代n次后停止
    patience = 5000
    patience_increase = 2
    improvement_threshold = 0.995
    validation_frequency = min(n_train_batches, patience / 2)
    best_validation_loss = numpy.inf
    test_score = 0.
    start_time = timeit.default_timer()
    done_looping = False
    epoch = 0
    while (epoch < n_epochs) and (not done_looping):    #设置最大迭代次数n_epochs
        epoch = epoch + 1
        for minibatch_index in xrange(n_train_batches):     #小批量梯度,实际中n_train)batches=83
            minibatch_avg_cost = train_model(minibatch_index)
            iter = (epoch - 1) * n_train_batches + minibatch_index    #minibatch_index最大只能取到82
            if (iter + 1) % validation_frequency == 0:         #满足这个条件说明完整的迭代了一个epoch,即将所有训练样本迭代了一次
                validation_losses = [validate_model(i) for i in xrange(n_valid_batches)]
                this_validation_loss = numpy.mean(validation_losses)   #计算验证集错误率
                print(epoch %i, minibatch %i / %i, validation error %f %% %(epoch, minibatch_index + 1, n_train_batches, this_validation_loss * 100.))
                if this_validation_loss < best_validation_loss:     #如果找到了优者
                    if this_validation_loss < best_validation_loss * improvement_threshold:  #如果找到了优者*0.995(即很吊的优者)
                        patience = max(patience, iter * patience_increase)   #遇到了更吊的优者就更新参数
                    best_validation_loss = this_validation_loss    #不论屌不屌,只要找到优者就存起来
                    test_losses = [test_model(i) for i in xrange(n_test_batches)]
                    test_score = numpy.mean(test_losses)    #计算测试集的错误率
                    print((epoch %i, minibatch %i / %i, test error of‘‘ best model %f %%) %(epoch,minibatch_index + 1,n_train_batches,test_score * 100.))
                    with open(best_model.pkl, w) as f:cPickle.dump(classifier, f)    #存储已跑完的训练模型结果!
            if patience <= iter:   #当满足这个条件跳出迭代!
                done_looping = True
                break
    end_time = timeit.default_timer()
    print((Optimization complete with best validation score of %f %%,‘‘with test performance %f %%) %(best_validation_loss*100.,test_score*100.))
    print The code run for %d epochs,with %f epochs/esc % (epoch, 1.*epoch/(end_time-start_time))

def predict():
    classifier = cPickle.load(open(best_model.pkl))   #调出已存好了的训练模型!
    predict_model = theano.function(inputs=[classifier.input], outputs=classifier.y_pr ed)  #预测模型
    dataset = data/mnist.pkl.gz
    datasets = load_data(dataset)
    test_set_x, test_set_y = datasets[2]
    test_set_x = test_set_x.get_value()      #取出其值
    #test_set_y = test_set_y
    predicted_values = predict_model(test_set_x[:10])        #取出前10个预测下
    print("predicted values for the first 10 examples in test set:")
    print predicted_values

 

Deep Learning Tutorial - Classifying MNIST digits using Logistic Regression