How to Split Off a Validation Set That's Closer to the Test Set
Whether you're competing in a contest, running experiments, or doing engineering work, we often run into situations where the training set and test set have different distributions. Typically, we split off a validation set from the training set and use it to tune hyperparameters (see The Meaning of Training Sets, Validation Sets, and Test Sets), for instance to control the number of training epochs so as to prevent overfitting. However, if the validation set itself differs substantially from the test set, then a model that performs well on the validation set doesn't necessarily perform well on the test set. So it's worth studying how to make the distribution of the validation set closer to that of the test set.
Two Scenarios
First, let's be clear about the scope of this post: we're considering the scenario where we can access the test set inputs themselves, but not the test set labels. If it's the kind of scenario where you submit your model for a closed evaluation and can't see the test set at all, there's not much we can do. Why would the test set and training set have mismatched distributions in the first place? There are mainly two scenarios. more
The first is a mismatch in the label distribution. That is, if we only look at the inputs $x$, the distribution is basically the same, but the corresponding $y$ distribution differs. A typical example is information extraction tasks: the training set is often constructed via "distant supervision + rough manual annotation," so it's large in scale but may contain quite a few errors and omissions, whereas the test set might be built through "repeated careful manual annotation," with very few errors. In this case, there's no way to build a better validation set simply by splitting the data differently.
The second is a mismatch in the input distribution. Put plainly, the distribution of $x$ differs, but the labeling of $y$ is basically correct in both cases. For example, in classification problems, the class distribution of the training set may differ from that of the test set; or in reading comprehension, the ratio of factual to non-factual questions in the training set may differ from the test set, and so on. In this case, we can adjust the sampling strategy appropriately so that the validation set's distribution matches the test set's more closely, so that validation results better reflect test set results.
A Discriminator
To achieve our goal, we label the training set as 0 and the test set as 1, and train a binary discriminator $D(x)$:
\begin{equation}-\mathbb{E}_{x\sim p(x)}[\log (1 - D(x))] - \mathbb{E}_{x\sim q(x)}[\log D(x)]\end{equation}
where $p(x)$ represents the distribution of the training set, and $q(x)$ that of the test set. Note that we're not mixing the training set and test set together for sampling and training; rather, we sample equal numbers of examples separately from the training set and the test set to form each batch — in other words, we need to oversample to balance the classes.
Some readers might worry about overfitting, i.e., that the discriminator perfectly separates the training set from the test set. In fact, when training the discriminator, we should also split off a validation set as in ordinary supervised training, and use it to decide the number of training epochs, which prevents severe overfitting. Alternatively, as is done in some examples online, one can simply use logistic regression as the discriminator, since logistic regression is simple enough that the risk of overfitting is much lower.
Similar to the discriminator in a GAN, it's not hard to derive that the theoretically optimal solution for $D(x)$ is
\begin{equation}D(x) = \frac{q(x)}{p(x)+q(x)}\label{eq:d}\end{equation}
In other words, once the discriminator is trained, we can treat it as encoding the relative magnitude of the test set distribution.
Importance Sampling
Whether optimizing a model or computing metrics, what we really want is to do so on the test set. That is, for a given objective $f(x)$ (e.g., the model's loss), what we want to compute is
\begin{equation}\mathbb{E}_{x\sim q(x)}[f(x)] = \int q(x) f(x) dx\end{equation}
But computing the objective $f(x)$ usually requires knowing the true labels of $x$, and since we don't know the test set's labels, we can't compute it directly. However, we do know the training set's labels, so we can resolve this via importance sampling:
\begin{equation}\int q(x) f(x) dx=\int p(x)\frac{q(x)}{p(x)} f(x) dx=\mathbb{E}_{x\sim p(x)}\left[\frac{q(x)}{p(x)} f(x)\right]\end{equation}
From formula $\eqref{eq:d}$, we know that $\frac{q(x)}{p(x)}=\frac{D(x)}{1-D(x)}$, so this ultimately becomes
\begin{equation}\mathbb{E}_{x\sim q(x)}[f(x)] = \mathbb{E}_{x\sim p(x)}\left[\frac{D(x)}{1-D(x)} f(x)\right]\label{eq:w}\end{equation}
Put simply, the idea behind importance sampling is to "pick out" from the training set those samples that resemble the test set, and assign them higher weight.
Final Strategy
From formula $\eqref{eq:w}$, we can derive two strategies:
The first is to directly apply the weighting from the formula. That is, we still split the training set and validation set via random shuffling, but assign each sample a weight $w(x)=\frac{D(x)}{1-D(x)}$. It's worth noting that a similar approach has already been used by some contestants in competitions, except that the weight commonly circulated is $D(x)$. I can't say for certain which is better, but from a theoretical derivation standpoint, $\frac{D(x)}{1-D(x)}$ should be the more principled choice.
The other strategy is to actually sample out the corresponding validation set. This isn't hard either: suppose all the samples in the training set are $x_1,x_2,\dots,x_N$; we normalize the weights as
\begin{equation}p_i = \frac{w(x_i)}{\sum\limits_{i=1}^N w(x_i)}\end{equation}
and then perform independent repeated sampling according to the distribution $p_1,p_2,\dots,p_N$ until we've sampled the desired number of examples. Note that this must be sampling with replacement, so the same sample may be drawn multiple times — and it should be kept with multiplicity in the validation set, not deduplicated, since deduplication would change the distribution.
Summary
This post approaches the discrepancy between the training set and test set from the angle of training a discriminator, and by combining this with importance sampling, we can obtain a validation set that's closer to the test set, or alternatively reweight the training samples so that the optimization process on the training set more closely matches the test set.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.