A brief note on unbiased and biased estimators

简述无偏估计和有偏估计 苏剑林 · 2019-06-19 · Mathematics statistics probability estimation

For most readers (including myself), the first biased estimator we ever encounter is probably the variance

\begin{equation}\hat{\sigma}^2_{\text{biased}} = \frac{1}{n}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2,\quad \hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i\label{eq:youpianfangcha}\end{equation}

and then we learn that the corresponding unbiased estimator should be

\begin{equation}\hat{\sigma}^2_{\text{unbiased}} = \frac{1}{n-1}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2\label{eq:wupianfangcha}\end{equation}

To many people, formula $\eqref{eq:youpianfangcha}$ looks like the reasonable one — how can it possibly be biased? And formula $\eqref{eq:wupianfangcha}$, which replaces $n$ with the counter-intuitive $n-1$, turns out to be the unbiased one instead?

Below I'll try to discuss, in as clear language as possible, the two concepts of unbiased and biased estimators.

Suppose we could sample an endless number of samples; then, in theory, the following estimate would be exact:

\begin{equation}\begin{aligned}\sigma^2 =&\, \mathbb{E}\left[(x - \mu)^2\right]=\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2\\ \mu =&\, \mathbb{E}[x]=\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n x_i\end{aligned}\end{equation}

This can also be understood as saying that, as the number of samples tends to infinity, biased and unbiased estimators become equivalent.

The problem is that in actual computation we can only sample a finite batch to compute the estimate — that is, $n$ is a fixed number. For instance, in stochastic gradient descent we use the average gradient over a batch of samples as an estimate of the gradient over the whole dataset. On the other hand, we don't just perform the estimation once and stop there — we may perform it many times: first we sample $n$ samples and compute once to get $\hat{\mu}_{1}$ and $\hat{\sigma}^2_{\text{biased},1}$, then randomly sample $n$ samples again to get $\hat{\mu}_{2}$ and $\hat{\sigma}^2_{\text{biased},2}$, and so on, obtaining $\left(\hat{\mu}_{3},\hat{\sigma}^2_{\text{biased},3}\right),\left(\hat{\mu}_{4},\hat{\sigma}^2_{\text{biased},4}\right),\dots$. What we want to know is:

\begin{equation}\begin{aligned}\mu &\xlongequal{?}\mathbb{E}\left[\hat{\mu}\right] = \lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^N \hat{\mu}_{i}\\ \sigma^2 &\xlongequal{?}\mathbb{E}\left[\hat{\sigma}^2_{\text{biased}}\right]=\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^N \hat{\sigma}^2_{\text{biased},i} \end{aligned}\end{equation}

In other words, does the "infinite average" of a "finite average" equal the average we actually want to compute?

As stated earlier, this article focuses on discussion and intuition rather than derivation, so I won't attempt a general proof. Instead, we'll use the simplest possible example: suppose $n=2$, i.e., when estimating with $\eqref{eq:youpianfangcha}$ or $\eqref{eq:wupianfangcha}$, we sample only two data points each time. In this case, the question we want to answer is:

\begin{equation}\begin{aligned}\mu &\xlongequal{?}\mathbb{E}_{x_1,x_2}\left[\frac{x_1 + x_2}{2}\right]\\ \sigma^2 &\xlongequal{?}\mathbb{E}_{x_1,x_2}\left[\frac{1}{2}\left(\left(x_1 - \frac{x_1 + x_2}{2}\right)^2 + \left(x_2 - \frac{x_1 + x_2}{2}\right)^2\right)\right] \end{aligned}\end{equation}

Since this case is simple enough, we can easily verify, for example, that

\begin{equation}\mathbb{E}_{x_1,x_2}\left[\frac{x_1 + x_2}{2}\right] = \mathbb{E}_{x_1}\left[\frac{x_1}{2}\right] + \mathbb{E}_{x_2}\left[\frac{x_2}{2}\right] = \frac{\mu}{2} + \frac{\mu}{2} = \mu\end{equation}

So the estimate of the mean obtained from two samples is indeed an unbiased estimator of the mean, and the same holds for more samples.

But the variance behaves differently:

\begin{equation}\begin{aligned}&\mathbb{E}_{x_1, x_2} \left[\frac{1}{2}\left(\left(x_1 - \frac{x_1 + x_2}{2}\right)^2 + \left(x_2 - \frac{x_1 + x_2}{2}\right)^2\right)\right]\\ =&\frac{1}{4}\mathbb{E}_{x_1, x_2} \left[\left(x_1 - x_2\right)^2\right]\\ =&\frac{1}{4}\mathbb{E}_{x_1, x_2} \left[x_1^2 + x_2^2 - 2 x_1 x_2\right]\\ =&\frac{1}{4}\Big(\mathbb{E}_{x_1} \left[x_1^2\right] + \mathbb{E}_{x_2} \left[x_2^2\right] - 2 \mathbb{E}_{x_1} \left[x_1\right] \mathbb{E}_{x_2} \left[x_2\right]\Big)\\ =&\frac{1}{4}\Big(2\mathbb{E}_{x} \left[x^2\right] - 2 \mu^2\Big)\\ =&\frac{1}{2}\Big(\mathbb{E}\left[x^2\right] - \mu^2\Big) \end{aligned}\end{equation}

Note that the exact expression for the variance should be $\mathbb{E}\left[x^2\right] - \mu^2$, so the two-sample quantity $\hat{\sigma}^2_{\text{biased}}$ is a biased estimator of the variance: even after repeating the estimation and averaging, it still underestimates the true variance. If we analyze the estimate based on $n$ samples, the leading factor turns out to be $(n-1)/n$. So we only need to multiply by $n/(n-1)$ to obtain an unbiased estimator of the variance, giving the final result $\eqref{eq:wupianfangcha}$.

Intuitively, when we use the finite-sample formula $\eqref{eq:youpianfangcha}$ to estimate the variance, because there are fewer samples, the fluctuation is also smaller, so the variance estimate tends to come out smaller — this is exactly what "biased" means here. In the extreme case, what if we sample just a single point for the estimate? Then the variance computed via formula $\eqref{eq:youpianfangcha}$ would be zero — and no matter how many times we repeat the experiment, the result stays zero. We certainly can't conclude that the variance of the entire population must be zero, right? This is the simplest possible example of a biased estimator.

Theoretically, the mechanism that produces bias is also easy to understand, because the formula for variance is equivalent to:

\begin{equation}\mathbb{E}\left[x^2\right] - \mathbb{E}\left[x\right]^2\end{equation}

Here, the expectation operator $\mathbb{E}$ is a linear operator, so the expression above is nonlinear in $\mathbb{E}$ (specifically quadratic, i.e., the $\mathbb{E}\left[x\right]^2$ term). Whenever an estimator is nonlinear with respect to the expectation operation $\mathbb{E}$ (note: the nonlinearity being emphasized here is with respect to the expectation operator, not with respect to the random variable itself), a naive finite-sample estimate is very likely to be biased. This is because composing a linear operation with another linear operation still yields a linear operation, but composing a linear operation with a nonlinear one no longer gives back the original nonlinear operation.

Not every biased estimator can be turned into an unbiased one simply by replacing $n$ with $n-1$, the way we did for variance. In general, the quantity we want to estimate may already be difficult to estimate at all, let alone determine whether that estimate is biased or unbiased. So removing the bias from a general estimator has to be handled case by case, depending on the specific problem at hand.