In Plain Terms: Models, and Picking Mangoes

漫话模型|模型与选芒果 苏剑林 · 2015-07-15 · Machine Learning model machine learning

Many people think that words like "model," "big data," and "machine learning" sound lofty and mysterious. In fact, they're not that different from picking fruit in everyday life. This post uses a few thousand words to try to teach everyone how to pick a good mango...

The Mango Analogy

Mango

Suppose I want to find the tasty ones out of a batch of mangoes. Since I can't just cut a mango open and taste it, all I can do is observe it — things like color, the smell of its skin, size, and so on are the quantities we can actually observe (features).

There are plenty of similar examples in daily life. Take buying matches (maybe young city dwellers these days have never even seen a matchbox?). How do you judge the quality of a box of matches? Surely you're not going to strike every single match to see if it lights? Obviously not — at most we can strike a few; if we struck them all, there'd be no matches left. But looking at how the matches look, and smelling them, that we can do.

more

We might notice that mangoes that are yellow and large tend to be sweet, but then we also notice that some mangoes that aren't very yellow, and are small, are also sweet. So exactly how much weight should color, smell, and size each carry? If we could figure out that weighting, we'd have found a way to predict whether a mango is tasty. The match example works the same way: we could strike a few matches, see which ones light, and from that summarize a method for predicting — without striking — which ones will light.

This is exactly what a model does. First we gather a batch of mangoes (the samples) and record their features (color, smell, size, etc.), then we have someone taste them and rate which ones are delicious and which aren't. From this batch of samples, we can then work out what weight color, smell, and size each carry. This summarizing process is what the machine does on its own.

Once this is done, we have a model that predicts whether a mango is tasty. It's a bit like a black box: from then on, you feed in the color, smell, size, and other data, and out comes the probability that it's delicious.

What Models Are For

From the analogy above, we can see that the most important thing a model does is solve two problems.

1. "One-size-fits-all" cutoffs

"One-size-fits-all" rules are the kind of thing that should really upset us. Think back to school, where a teacher, without asking any questions, simply "sentenced you to death" — that kind of crude, blanket judgment is a textbook case of a one-size-fits-all cutoff. Such rules do have some level of accuracy, but not every problem can be solved this way. Worse, a one-size-fits-all cutoff often ends up misclassifying the very best specimens!

For example, suppose I want to find the top students in my class. Naturally we'd think that academic performance is proportional to time spent studying, so we might decide that "studying more than 5 hours a day" identifies the good students. That's a one-size-fits-all rule. But there are clearly people who are simply gifted, or study very efficiently, who only need one hour a day to get excellent grades. Such people get wrongly "cut off" by our rule — and, notably, it's exactly the best specimens that get cut.

2. Automated learning

Fruit

Let's go back to the mango example. Suppose that, through "years of experience," we've worked out a method for judging tasty mangoes even without a model. At this point someone might say, "What's so great about your model? We can do just as well ourselves." But suppose I no longer want to eat mangoes — now I want oranges, or grapes. How do we then predict how tasty oranges or grapes are? We surely can't wait years and years to accumulate "years of experience" with lychees and apples all over again, can we? That would cost enormous time, not to mention manpower.

Of course, other people might already have experience with oranges or grapes, and we could ask them. But asking has its own cost — just think of all the paid training programs cropping up everywhere, and you'll see what I mean.

This is exactly the problem a model solves. It lets us start from an existing batch of samples (whether mangoes, apples, or lychees) and automatically, mechanically "summarize" (a process we call learning) a method of judgment. Since the learning is all done by the machine, it saves us a great deal of effort. We just need to make a cup of tea and wait for the model's results, then check whether they're any good. That's certainly better than having to learn and summarize the rules ourselves, and then also having to judge how well we learned them.

How to Build a Model

Building a good model generally involves the following steps.

1. Preparing the samples

The samples are that batch of "mangoes" we use to learn from.

In fact, the process of building a model is quite similar to how humans learn. If we asked a human to do this, they would surely first gather some mangoes, record their color, size, smell, and other features, then cut them all open and taste them, note which are sour and which are sweet, and finally summarize the pattern.

For a model, the model takes over the summarizing process — that is, the last step. The preparatory work before that still needs to be done by us. We have to taste a batch of mangoes ourselves, record information about that batch, and then feed all that information into the model. The model can then learn automatically, and once it has learned, we can use it to predict the taste of new mangoes.

Preparing samples means preparing both good and bad samples. In other words, you need to find a batch of tasty mangoes and record their features, and also find a batch of bad-tasting mangoes and record their features, and then tell the model all of this information so it can learn on its own. In this process, the human's role is that of a record-keeper.

2. Preparing the features

Features are the variables related to the outcome we're trying to judge — they're the basis on which the model makes its predictions.

Simply put, features are the "what" in "what is mango tastiness related to." If we believe that how tasty a mango is depends on its size, color, and smell, then "size," "color," and "smell" are the model's features — provided, of course, that this information can be quantified.

Features come in good and bad varieties. Good features help the model make accurate predictions, while bad features are, at best, of no help to prediction. For instance, which tree a mango was picked from, or what day of the week it was picked, are probably not good features — that is, this information usually doesn't help us judge how tasty a mango will be. (Note the word "usually" — this isn't absolute. Maybe mangoes picked from tree A really are tastier than those from tree B, who knows.)

Good features are absolutely critical to a model. In fact, you could say that finding good features — whether by hand or by machine — is the single most important part of modeling. A good data scientist, in the process of building a model, should devote most of their effort to selecting features. Yet these days, most practitioners fall into the trap of pouring the bulk of their effort into the model itself (that is, step 3 below).

3. Preparing the model

Preparing the model really just means choosing a model — that is, deciding what method to use to learn. This is analogous to how different people have different learning methods and experiences, and the question is which method to adopt.

In the actual field of machine learning, there are quite a number of models — for instance, linear models versus non-linear models. Linear models include logistic regression, SVM, and so on; non-linear models include random forests, GBDT, neural networks, and more. When it comes to models, there are a few general things worth keeping in mind:

(1) The model is not the most important thing.
In fact, the most important part of modeling is choosing the right features. Once you've picked good features, the differences between models won't matter that much. So don't put most of your effort into choosing a model.

(2) Guard against overfitting.
Overfitting is a phenomenon that's fairly hard to detect. Broadly speaking, it means the resulting model performs beautifully on the in-sample test but falls apart completely in real-world use. Common ways to prevent overfitting are to introduce a regularization coefficient (i.e., a penalty term), or to limit the depth (for tree-based models).

(3) Prefer linear models where possible.
Non-linear models, like GBDT, generally perform quite well, but they're also more prone to overfitting. So unless a non-linear model does substantially better than a linear one, it's best to go with the linear model, since it tends to be more stable. This principle is really just an application of Occam's razor: "Do not multiply entities beyond necessity."

Last but Not Least

Whatever else is said, one thing needs to be emphasized: models are useful, but they are not omnipotent, nor are they the most important thing. Don't be so devoted to models that you lose your own initiative and judgment. A model can be considered a work of art — but only on the condition that you are an artist.