How to Find an Unbiased Estimator

An unbiased estimator is a statistic whose expected value equals the quantity it is intended to estimate. To find one, you need the sampling model, the target parameter (or function of a parameter), and the observations available. Without those details there is no single estimator to calculate.

The general method is to propose a statistic based on the data, calculate its expectation under the model, and check whether that expectation equals the target for every allowed parameter value. A familiar example is the sample mean, which is unbiased for a population mean under independent, identically distributed sampling with a finite mean. A Bernoulli example below shows how to correct a biased estimator.

Definition of unbiasedness

Let $X$ denote the observed data, with a probability distribution depending on an unknown parameter $\theta$. An estimator $T(X)$ is unbiased for a target $g(\theta)$ if

for every $\theta$ in the parameter space. If the target is the parameter itself, then $g(\theta)=\theta$.

The expectation is taken over repeated samples from the model, with the parameter held fixed. Unbiasedness is therefore a property of an estimator and a model together, not a property of a formula by itself. An estimator can be unbiased for one target and biased for another.

A practical method

1. State the model and target. For example, specify that $X_1,\ldots,X_n$ are independent Bernoulli observations with success probability $p$, and that the target is $p(1-p)$.
2. Propose a statistic. Use a function of the observations, possibly with constants that depend on the sample size but not on unknown parameters.
3. Calculate its expectation. Use the probability model and properties such as linearity of expectation. If you need to choose a correction factor, set the expectation equal to the target and solve for it.
4. Verify the result. Check the equality for all permitted parameter values and confirm that the estimator has a finite expectation.

If a candidate has the wrong expectation, it may still be possible to adjust it. For instance, if $\mathbb{E}_\theta[T]=a g(\theta)$ for a known constant $a\ne 0$, then $T/a$ is unbiased for $g(\theta)$. If the expectation cannot be made equal to the target using an available statistic, an unbiased estimator may not exist for that model and sample size.

Worked example: estimating Bernoulli variance

Suppose $X_1,\ldots,X_n$ are independent Bernoulli random variables, each equal to 1 with probability $p$ and 0 otherwise. The variance of each observation is $p(1-p)$. A natural plug-in estimate replaces $p$ with the sample proportion

But plugging an unbiased estimator into a nonlinear function does not generally produce an unbiased estimator. In this case,

Thus $\bar X(1-\bar X)$ is biased downward when $n>1$. Multiply it by $n/(n-1)$ to obtain an unbiased estimator:

Indeed, its expectation is $p(1-p)$. This estimator is also the sample variance computed with denominator $n-1$:

For example, if $n=4$ and the observed values are $1,0,1,1$, then $\bar X=3/4$ and the estimate is $(4/3)(3/4)(1/4)=1/4$. The value from this particular sample need not equal the true variance; unbiasedness means that the estimator averages to the true variance over repeated samples.

A simpler example: estimating a mean

If $X_1,\ldots,X_n$ are identically distributed with finite mean $\mu$, then linearity of expectation gives

So the sample mean is unbiased for the population mean. Independence is not needed for this expectation calculation, though it matters for other properties such as the variance of $\bar X$.

Common mistakes

• Using a statistic without specifying its target. The same statistic can be unbiased for one quantity and biased for another.
• Assuming a function of an unbiased estimator is unbiased. In general, $\mathbb{E}[g(T)]$ is not equal to $g(\mathbb{E}[T])$. The Bernoulli plug-in example demonstrates this.
• Checking only one parameter value. Unbiasedness requires the expectation identity to hold throughout the parameter space.
• Confusing unbiasedness with accuracy in one sample. An unbiased estimate can differ substantially from the true value in a particular dataset. Unbiasedness describes its long-run average, not its error in every sample.
• Assuming an unbiased estimator always exists. Its existence depends on the model, target, and sample size. For example, with just one Bernoulli observation, no statistic can be unbiased for $p(1-p)$: the expectation of any statistic is linear in $p$, whereas $p(1-p)$ is quadratic.