What Do Training, Validation, and Test Sets Really Mean?

训练集、验证集和测试集的意义 苏剑林 · 2017-10-14 · Machine Learning model deep learning

In supervised machine learning, we constantly talk about the training set, the validation set, and the test set. The distinctions among these three can be confusing—in particular, quite a few readers are unclear on what actually separates the validation set from the test set.

Splitting the Data

If we already have a large labeled dataset in hand and want to build and evaluate a supervised model, we typically split it via uniform random sampling into a training set, a validation set, and a test set. These three sets must not overlap, and a common ratio is 8:1:1—though of course that ratio is just a convention chosen by humans. From this point of view, all three sets follow the same underlying distribution.

If instead we're working on a competition, the organizers usually only provide a labeled dataset (to serve as the training set) plus an unlabeled test set. In that case, when building our model, we typically carve out a validation set manually from the training set ourselves. In this scenario we generally don't split out a separate test set as well, for two likely reasons: (1) competition organizers tend to be rather stingy, so the training set is already small to begin with; and (2) we have no way of guaranteeing that the test set we ultimately submit against follows exactly the same distribution as the training set, so splitting out yet another test set that's distributed like the training set wouldn't be very meaningful anyway.

Parameters

Once we have a model, the training set is used to train its parameters—more precisely, it's what drives gradient descent. The validation set, on the other hand, is generally used at the end of each epoch to check the current model's accuracy. Since the validation set has no overlap with the training set, this accuracy measurement is trustworthy. So why would we still need a test set on top of that?

To answer this, we need to distinguish between different kinds of parameters in a model. In fact, for any given model, its parameters can be split into ordinary parameters and hyperparameters. Setting reinforcement learning aside, ordinary parameters are the ones updated by gradient descent—that is, the parameters that the training set updates. Hyperparameters, on the other hand, include things like the number of layers, the number of units per layer, the number of training iterations, the learning rate, and so on—these fall outside the scope of gradient-descent updates. Although there now exist algorithms for automatically searching over a model's hyperparameters, in most cases we still tune them by hand, guided by the validation set.

Therefore

That means: in the narrow sense, the validation set never participates in the gradient descent process—it is, strictly speaking, never "trained on." But in a broader sense, the validation set does participate in a process of "manual hyperparameter tuning": we adjust the number of iterations, the learning rate, and so on based on validation results, so as to make the outcome optimal on the validation set. In that sense, we could also say the validation set has, in effect, participated in training.

Given that, it becomes clear that we still need a set that has genuinely never been touched during training in any way—that is the test set. We neither run gradient descent on the test set nor use it to control hyperparameters; we only use it, once training is completely finished, to measure the final accuracy.

And Yet

Sharp readers will immediately spot the analogy: this could go on forever. If accuracy on the test set turns out poor, we'll likely go back and adjust the model's various parameters again—at which point the test set, too, could be said to have participated in training. In that case, we might need a "test-test set," and perhaps after that a "test-test-test set" ...

Let's not go there. Let's just agree to stop at the test set.