Chatting About Neural Networks and Deep Learning

闲聊：神经网络与深度学习 苏剑林 · 2015-06-06 · Machine Learning neural network deep learning machine learning

Neural Network

Among all machine learning models, perhaps the most interesting and profound is the neural network. I'd like to try my hand at discussing it too. Of course, this post isn't meant to be an introduction to neural networks from scratch—it's just me sharing my personal understanding of neural networks. Readers who want a more thorough grounding in neural networks and deep learning should check out the following tutorials:

http://deeplearning.stanford.edu/wiki/index.php/UFLDL Tutorial

http://blog.csdn.net/zouxy09/article/details/8775360

Machine Classification

Let's use classification as our example. In data mining and machine learning, there are many classification problems—for instance, classifying the sentiment of a sentence, which could be coarsely divided into "positive" or "negative," or more finely into happy, angry, sad, and so on. Another classic classification problem is handwritten digit recognition, where images are sorted into 10 classes (0, 1, 2, 3, 4, 5, 6, 7, 8, 9). Consequently, a great many classification models have been developed. more

At its core, a classification model is doing fitting—the model is really just a function (or a family of functions) with some undetermined parameters. Based on the available data, we define a loss function (the most common loss function is the sum of squared errors; readers unfamiliar with this can recall the least squares method), and then we optimize the loss function to be as small as possible, thereby solving for the undetermined parameter values. Once we have the parameter values, we can use this function to make predictions. That's the basic idea behind classification; details like preventing overfitting are secondary concerns we won't discuss here.

This idea seems simple enough, but it hides two fundamental, hard-to-solve problems: 1. What are the independent variables of the function? 2. What is the function itself? In other words, how do I know which things (features) are useful for the classification task I care about? And second, complex nonlinear phenomena are everywhere—once I've found these features, how do I know which function to use to fit them? In truth, neither of these two questions has ever really had a good answer. Before deep learning came along, the choice of model and features was essentially done by hand. In other words, after decades of development, the field of machine learning still hadn't resolved these two fundamental problems!

The emergence of deep learning brought real hope for solving both. And the foundation of deep learning is the neural network.

Neural Networks

Neural networks address the second problem: what is the function? Traditional models, such as linear regression and logistic regression, essentially require us to manually specify the form of the function. But there are so many nonlinear functions out there that simply picking a few off-the-shelf ones often gives limited fitting performance—and how well the fit works depends heavily on finding good features, which brings us back to the still-unsolved first problem: what are the independent variables of the function? (For example, suppose a function is $y=x^2+x$, a quadratic nonlinear function. If we tried to fit it with linear regression, the result would inevitably be poor. But if I define a new feature (independent variable) $t=x^2$, then $y=t+x$ becomes a linear function of $t,x$, and now a linear model can handle it perfectly well. The problem is: without knowing the specific form of $y$, how would we ever find the feature $t=x^2$? This is largely a matter of experience and luck.)

To solve the "what is the function" problem, there are various approaches one could imagine. For instance, we've all studied Taylor series and know that general nonlinear functions can be approximated via Taylor expansion. So a natural idea is: why not use a high-degree polynomial to approximate it? A high-degree polynomial is indeed a good idea, but anyone who has studied numerical methods probably knows that polynomial fitting tends to fit the training data very well but perform poorly on test data—that is, it's prone to overfitting. So, is there some other way? Yes! The person who first proposed the neural network said—use composite functions to do the fitting!

That's right, neural networks fit data through repeated function composition! And it's the simplest possible composition: one is a linear function, and the other is the simplest possible nonlinear function: the binary function $\theta(x)$

$$\theta(x)=\left\{\begin{aligned}1,&\,x\geq 0\\-1,&\,x < 0\end{aligned}\right.$$

Let's take the simplest three-layer neural network for binary classification as an example:

Neural network diagram

First, we take a weighted linear combination of the input features (with weights $w^{(1)}_{j,i}$), then apply $\theta(x)$ to the result of this combination in the hidden layer. Next, we take a weighted linear combination of that result (with weights $w^{(2)}_{k,j}$), and finally apply $\theta(x)$ once more at the output layer to produce the output. The whole process consists of two linear combinations and two applications of $\theta(x)$, alternating with each other. As we can see, the entire process is a doubly-composed function:

$$\theta\left(\sum_{j} w^{(2)}_{k,j} \theta\left(\sum_{i} w^{(1)}_{j,i} x_i\right)\right)$$

where the nonlinear function $\theta(x)$ is called the activation function. Next we need to define a loss function—that is, the function we want to minimize through optimization. However, the optimization process involves taking derivatives, and $\theta(x)$ is discrete, so it can't be differentiated. For this reason, we typically replace it with some continuous activation function instead, such as the sigmoid function

$$S(x)=\frac{1}{1+e^{-kx}}$$

Now the question is: does the model proposed above actually possess the kind of power we imagine it to have? The following passage illustrates just how powerful neural networks really are:

In 1943, the psychologist W.S. McCulloch and the mathematical logician W. Pitts constructed the neural network described above along with a corresponding mathematical model, known as the MP model. Through the MP model, they proposed a formal mathematical description of neurons and a network architecture, and proved that a single neuron could carry out logical functions. It's worth noting that being able to perform logical functions means being able to accomplish everything that today's computers can do—after all, modern computers are ultimately just combinations of a series of logical instructions. (Of course, there's the question of speed here—but setting speed aside, the MP model can indeed accomplish everything that present-day computers can accomplish.)

Deep Learning

What we've discussed above is just a doubly-composed function. Readers will readily think: since this approach works, wouldn't triply-, quadruply-, or even more highly-composed functions work even better? Indeed—that's precisely the original idea behind deep learning. After all this build-up, we've finally arrived at deep learning.

Neural networks, including multi-layer neural networks, are not a recent invention—in fact, these models were proposed decades ago, along with some algorithms for solving them. So what really distinguishes deep learning from earlier neural networks? A common answer is that deep learning networks tend to have many more layers (five or six hidden layers is quite common), and they employ more effective solving algorithms that better address issues like gradient divergence and convergence to local minima. But I don't think that's the essential distinction. In my view, the difference between deep learning and traditional neural networks is that deep learning is devoted to solving the first problem we mentioned earlier: what are the independent variables of the function?

In other words, deep learning algorithms are designed to discover good features—a task that used to be done entirely by hand, with no guarantee that the chosen features were actually "good." Now, machines can accomplish this automatically (in an unsupervised fashion), and the results are no worse than those achieved through manual feature selection. Once good features are found, even a linear model can perform quite well. This is precisely why deep learning is said to have pushed the field of artificial intelligence forward by a great leap, and why "deep learning" is regarded as the algorithm that comes closest to true artificial intelligence today.

Why is it so important to find good features? For several reasons:

First, accuracy. Finding good features means effectively filtering out confounding factors, so that everything fed into the model is genuinely useful, reducing redundancy.
Second, computational cost. Consider image classification, where an image has 1000×1000 pixels. If we feed all the pixels in directly, that's 10⁶ features. Feeding all of these into a deep learning model could result in as many as 10¹² parameters or more—solving for a model with that many parameters is essentially infeasible. Finding good features means feeding only the useful information into the model, thereby reducing the parameter count and making the problem tractable.
Third, storage. Good feature-extraction algorithms can also serve as effective file compression, reducing both storage requirements and the memory needed during training.

Autoencoders

How does deep learning achieve all this? In four words: information loss!

First, we need to recognize that no matter how intelligent our algorithm is, it's ultimately built to accomplish some specific task in some specific domain. This means that the raw data is bound to contain information we don't need—and it's entirely possible that the unneeded information outnumbers the needed information. Wouldn't it be great if we could strip away this unneeded information and keep only what's useful? (Information loss—ideally, what gets lost is precisely the information we don't need.) You may not have realized it, but our own brains are constantly doing exactly this. We call that mental process—abstraction!

That's right, abstraction—abstraction is a process of information loss. For example, take a pile of balls: basketballs, soccer balls, volleyballs, ping-pong balls, all sorts of balls. Our first-level understanding of them is: they're all (approximately) "spheres"—sets of points equidistant from some center! But don't forget, these are a bunch of very different balls—their sizes, colors, materials, and so on all differ. By recognizing them all as "balls," we've effectively discarded information about size, color, material, and so forth, keeping only the shared feature of this pile of balls: their shape is roughly spherical. This process is exactly our abstract thinking, and it's also a process of information loss!

How does deep learning achieve this? The "autoencoder" is one of its core techniques, and it too is built on neural networks. Consider the following three-layer neural network:

Autoencoder diagram

Here, the network's input and output have the same dimensionality (both 100-dimensional), while the hidden layer in between has only 50 nodes. The autoencoder aims to train a simple function $x=x$ via the network above, such that the input and output end up being the same. However, during training, going from the input layer to the hidden layer, the dimensionality drops from 100 to 50—meaning some information is necessarily lost. Yet going from the hidden layer back to the output layer, the dimensionality is restored to 100. Since information has already been lost, in principle this reconstruction process ought to be impossible.

And yet, we deliberately force the network to be trained this way—so what happens? The machine has no choice but to reconstruct the original data as best it can from these 50 dimensions. To make the reconstruction as good as possible, the machine is forced to extract the common features shared across the whole batch of input data, and use those as the basis for reconstruction. As a simple example, consider the following four images:

Letter-background

These are four different images—how does the reader tell them apart? Clearly, the backgrounds of all four images are identical; what differs is the letter overlaid on top. If I asked you to memorize the difference between these images in a short amount of time, you would only be able to remember the individual letters. But keep in mind, these are four complete images, and yet all we remember is four letters—we've discarded the background entirely, losing the vast majority of the information!

But if our task is simply to "identify the letter in the image," then discarding the background is entirely correct! Once the background is removed, what remains is exactly the good feature we need for making our judgment—the background, if kept, would actually be a source of interference.

This is precisely the autoencoding process—it's a process of information loss, and it's a process of abstraction!

Summary

Of course, deep learning is an extraordinarily rich subject with all kinds of variations. It's worth noting that many deep learning–based models have achieved State-of-the-Art results (i.e., the most advanced results currently known), which speaks to just how powerful deep learning really is.

This post is not an introductory tutorial on deep learning—it's merely a sketch of my own rudimentary understanding of neural networks and deep learning. If readers want to systematically study the underlying theory, it's best to first read some books on data mining and machine learning to get familiar with the basic concepts, and then go through the two articles mentioned at the beginning of this post.

http://ufldl.stanford.edu/wiki/UFLDL Tutorial

http://blog.csdn.net/zouxy09/article/details/8775360