The idea of deep learning could be explain by graph_00
The first deep learning algorithm introduced by UFLDL is sparse autoencoder, the implementation details of sparse autoencoder is quite daunting, even it may be the most easiest algorithm to understand in the deep learning algorithms.
At the first glance, sparse autoencoder looks quite similar to the traditional neural network, the different part are
- It is train layer by layer
- The dimension of the hidden layers usually less than the input layers
- Sparse autoencoder add sparsity constraint on the hidden unit(autoencoder do not have this constraint)
- The purpose of sparse autoencoder is find out how to use less features to reconstruct the input
The meaning of train it layer by layer is that we can use the output of the hidden layer as the next input of the autoencoder layer(this autoencoder also contain three layers). By this way we can build efficient deep network.
The implementation details of sparse autoencoder of mine can be found at here, the example and the images file. graph_02 show the results train by my implementation. Since the mini-batch do not work well in this case, I need to use the full batch size to do the gradient descent, without algorithm like L-BFGS, the speed could be quite slow. In the next post, I will use shark to recognize the hand written digts from MNIST.
Atlas is difficult to build on windows and is not well optimized for Windows 64 bit, if you want to get maximum speed from shark, better train the data under unix/linux.