Use neutral network for unsupervised learning

14 Oct

UFLDL_Tutorial

Main idea:

Neural net can be used for predictive modeling: train a model with
$(x_1, y_1), ... , (x_n , y_n )$
, then we can use the model to predict y for a new x.

Now if in our training data, we have
$x_1 = y_1, ... , x_n = y_n$
, for example, if x is an image, y is the same image, then the trained neural network is simply to predict an identity function. Seems pretty boring, right? Not necessary, if our neutral net is a sparse model (i.e., most parameters are zero or most nodes are inactivated), then the neutral net actually output a sparse representation (i.e coding) of x.

This tutorial gives interesting examples in image and audio coding.

For this sparse coding to work well, it is expected that there must be some structures in the training data, which is true for natural images and human voices.