What Is an Unbiased Estimator of a Bernoulli Distribution?

If you observe repeated yes/no outcomes and want to estimate the probability of “yes,” use the proportion of yes outcomes. Under the usual assumption that the observations are independent and have the same Bernoulli distribution, that proportion is an unbiased estimator of the unknown probability.

Bernoulli observations and unbiasedness

A Bernoulli random variable $X$ takes the value $1$ with probability $p$ and $0$ with probability $1-p$. Its expected value is

An estimator is a statistic calculated from data. It is unbiased for a parameter when its expected value, taken over repeated samples, equals that parameter. Thus, if $X$ is one Bernoulli observation, $X$ itself is already an unbiased estimator of $p$. With multiple observations, averaging them usually gives a more informative estimate.

The sample proportion is unbiased

Suppose $X_1,\ldots,X_n$ are independent Bernoulli observations with the same unknown probability $p$. Define the sample mean

By linearity of expectation, each observation has expected value $p$, so

Therefore, $\widehat{p}$ is unbiased for $p$. Independence is not needed for this expectation calculation; having the same probability $p$ for each observation is enough. Independence does matter for the usual variance calculation and other properties of the estimator.

If $K=\sum_{i=1}^{n}X_i$ is the number of successes, the same estimator can be written as

For independent, identically distributed observations, $K$ has a binomial distribution with parameters $n$ and $p$.

Worked example

Suppose you record eight outcomes, coding a success as 1 and a failure as 0, and observe five successes. Then $n=8$ and $K=5$, giving

The estimate from this particular sample is 0.625. Unbiasedness does not mean that this estimate must equal the true probability in every sample. It means that if you repeatedly took samples of eight observations and calculated the proportion each time, the average of those proportions would be $p$.

The estimator’s variance under the independent, identical Bernoulli model is

This shows why increasing the sample size generally reduces the estimator’s variability. The variance depends on the unknown $p$, but its maximum possible value is $1/(4n)$, attained at $p=1/2$.

If you mean the whole distribution

A Bernoulli distribution is fully specified by its probability of 1; the probability of 0 is then $1-p$. Estimating $p$ therefore estimates the whole distribution. In terms of observed data, the empirical estimates are

Both are unbiased for the corresponding probabilities: the first has expectation $p$, and the second has expectation $1-p$. This is also an example of the empirical distribution assigning each possible outcome its observed sample frequency.

Common mistakes

• Using the number of successes as the estimate: $K$ has expectation $np$, not $p$. Use $K/n$ to estimate $p$ when the sample size is known. The exception is a single observation, where $n=1$ and $K=X_1$.
• Confusing unbiasedness with certainty: An unbiased estimator can be above or below the true value in a particular sample. The property concerns its average over repeated samples.
• Treating unbiasedness as a guarantee of precision: An estimator may be unbiased but variable. The sample size affects the variance, so unbiasedness alone does not tell you how close one estimate is likely to be to $p$.
• Ignoring the outcome coding: The sample mean estimates the probability of whichever outcome is coded as 1. If you reverse the coding, it estimates the probability of the other outcome.

For independent Bernoulli data, use the sample proportion of successes to estimate the success probability. It is unbiased because its expected value is the true probability.