How to Apply Bayes’ Theorem to Multiple Random Variables

Bayes’ theorem extends directly to several unknown random variables: use their joint prior and the likelihood of the evidence given all of them. To find the posterior for just one variable, sum or integrate the joint posterior over the variables you do not need. The variables do not have to be independent.

The joint form of Bayes’ theorem

Let $X_1,\ldots,X_n$ be unknown variables and let $E$ denote observed evidence. For discrete variables,

$$P(X_1=x_1,\ldots,X_n=x_n\mid E) =\frac{P(E\mid X_1=x_1,\ldots,X_n=x_n)P(X_1=x_1,\ldots,X_n=x_n)}{P(E)}.$$

The prior in the numerator is the joint probability of the variables taking those values. In general, it is not the product of their individual probabilities. The denominator is the probability of the evidence under all possible values of the unknowns:

$$P(E)=\sum_{x_1,\ldots,x_n} P(E\mid x_1,\ldots,x_n)P(x_1,\ldots,x_n).$$

The sum ranges over every possible combination of values. For continuous variables, replace the sum with an integral and use probability densities; if the evidence is continuous, use its density as well. For example, with two continuous unknowns $X$ and $Z$,

$$p(x,z\mid e)=\frac{p(e\mid x,z)p(x,z)}{p(e)}, \qquad p(e)=\int\!\int p(e\mid x,z)p(x,z)\,dx\,dz.$$

Here $p(x,z)$ is the joint prior density. The posterior density integrates to 1, just as the discrete posterior probabilities sum to 1.

Finding the posterior for only one variable

Often you want to infer $X$ but do not care about another unknown variable $Z$. First form the joint posterior, then marginalize over $Z$:

$$P(X=x\mid E)=\sum_z P(X=x,Z=z\mid E)$$

for discrete $Z$. For continuous $Z$, use an integral instead:

$$p(x\mid e)=\int p(x,z\mid e)\,dz =\frac{\int p(e\mid x,z)p(x,z)\,dz}{p(e)}.$$

This step accounts for every possible value of the nuisance variable rather than pretending it is known. If there are several nuisance variables, sum or integrate over all of them.

Worked example with two binary variables

Suppose $A$ and $B$ are binary unknowns, and an observation $E$ has the following prior probabilities and likelihoods. The prior entries already describe the joint distribution, so no independence assumption is needed.

$A$	$B$	Prior $P(A,B)$	Likelihood $P(E\mid A,B)$	Product
0	0	0.50	0.10	0.050
0	1	0.20	0.20	0.040
1	0	0.20	0.60	0.120
1	1	0.10	0.90	0.090

The last column is the likelihood multiplied by the prior for each combination. Add those products to get the evidence probability:

$$P(E)=0.050+0.040+0.120+0.090=0.300.$$

Divide each product by $0.300$ to get the joint posterior:

$A$	$B$	Posterior $P(A,B\mid E)$
0	0	$0.050/0.300=1/6$
0	1	$0.040/0.300=2/15$
1	0	$0.120/0.300=0.40$
1	1	$0.090/0.300=0.30$

For the posterior probability that $A=1$, add the rows with $A=1$:

$$P(A=1\mid E)=0.40+0.30=0.70.$$

For $B=1$, add the rows with $B=1$:

$$P(B=1\mid E)=\frac{2}{15}+0.30\approx 0.433.$$

These marginal results come from the joint posterior. The same method works with more variables, although the number of combinations can grow quickly.

When can the likelihood be split?

If the evidence consists of several observations $E_1,\ldots,E_m$, the likelihood is generally the joint probability $P(E_1,\ldots,E_m\mid X_1,\ldots,X_n)$. You may write it as a product of separate likelihoods only when the observations are conditionally independent given the unknown variables. Similarly, replacing a joint prior with a product of individual priors requires the unknown variables to be independent under that prior.

A model can have some conditional independence relationships and not others. Use only the independence assumptions justified by the model or the problem statement; Bayes’ theorem itself does not supply them.

Common mistakes

• Multiplying marginal priors automatically. In general, $P(X,Z)$ is not $P(X)P(Z)$. Use the joint prior unless independence is established.
• Forgetting the denominator. The likelihood times the prior gives unnormalized posterior weights. Divide by the total evidence probability to normalize them.
• Ignoring nuisance variables. If a variable is unknown but not of interest, marginalize over it rather than fixing it to an arbitrary value.
• Confusing densities with point probabilities. For continuous variables, a density at a point is not the probability of that exact value. Bayes’ density formula gives a conditional density, and the evidence density must be positive.

For discrete variables the rule is a sum over possible values; for continuous variables it is an integral. In either case, the procedure is the same: combine the joint prior with the likelihood, normalize, and marginalize if you need a smaller posterior distribution.