Conditional Probability and Bayes Theorem

01 / UNDERSTAND & APPLY

1. Probability belongs to a specified experiment

A probability model begins with an experiment, a sample space Ω of possible outcomes, and probabilities assigned to events. A statement such as “choose randomly” is incomplete unless the selection rule is understood. Uniformly choosing a box and then an item is generally different from uniformly choosing an item from all boxes combined, especially when the box sizes differ.

An event is a subset of outcomes. For a finite equally likely sample space, P(A) = |A|/|Ω|. Equal likelihood is a condition, not a consequence of having listed some possibilities. For two fair independent dice, the 36 ordered face pairs are equally likely; the eleven possible sums from 2 through 12 are not, because different sums have different numbers of underlying pairs.

Probabilities satisfy P(Ω) = 1, P(A) ≥ 0, and additivity for disjoint events. These rules imply 0 ≤ P(A) ≤ 1 and P(Aᶜ) = 1 − P(A). In the finite equally likely models used for the counting examples here, a probability-zero event contains no outcomes. In more general continuous models, probability zero need not mean logically impossible, so avoid extending that finite intuition without qualification.

Before calculating, write the event in words and symbols, state the selection mechanism, and decide whether order matters in the elementary outcomes. A technically correct fraction formed from the wrong experiment answers a different question. Clear modelling is the main safeguard against this kind of error.

02 / UNDERSTAND & APPLY

2. Event algebra: “or” includes the overlap

The union A∪B means A or B or both. The intersection A∩B means both. The complement Aᶜ means not A. The addition rule is P(A∪B) = P(A) + P(B) − P(A∩B), because adding the two individual probabilities counts the overlap twice. If A and B are disjoint, the overlap is empty and the subtraction term is zero.

Exactly one of A,B is the event (A∩Bᶜ)∪(Aᶜ∩B). Its probability is P(A) + P(B) − 2P(A∩B). At least one is the union, while neither is its complement. Translating these phrases correctly often matters more than any later arithmetic.

For three events, inclusion-exclusion adds the three single probabilities, subtracts the three pairwise intersections, and adds the triple intersection back. The alternating corrections account for how many times each type of outcome was counted. Use a Venn-region partition to understand the signs rather than memorizing them in isolation.

Probability bounds provide a useful audit. The intersection lies between max(0,P(A)+P(B)−1) and min(P(A),P(B)). The union lies between max(P(A),P(B)) and min(1,P(A)+P(B)). If supplied or computed values violate these bounds, no probability model can realize them as stated.

03 / UNDERSTAND & APPLY

3. Choose one consistent level of outcomes

If r distinct objects are selected without replacement in an order that matters, the number of equally likely ordered samples is n(n−1)⋯(n−r+1). If only the selected set matters and every r-subset is equally likely, the count is C(n,r). Either model can give the same probability, but the numerator and denominator must use the same convention.

In an urn with colored balls, the balls are distinct elementary objects even if only their colors are reported. Different color patterns need not be equally likely. Labeling the balls mentally prevents treating “two red,” “one of each,” and “two blue” as three equally likely outcomes when their underlying counts differ.

For a sample of size r from R red and B blue balls without replacement, the probability of exactly k red is C(R,k)C(B,r−k)/C(R+B,r), with feasible integers k satisfying 0 ≤ k ≤ R and 0 ≤ r−k ≤ B. The numerator chooses the required red and blue subsets independently from disjoint groups; the denominator counts all uniformly chosen r-subsets.

With replacement, the same physical ball may be drawn again and category probabilities remain fixed when draws are independently randomized. Without replacement, later probabilities depend on earlier draws. These experiments have different sample spaces and different denominators, so state the replacement rule before using a formula.

04 / UNDERSTAND & APPLY

4. Count the complement when the event has many cases

At least one is often easier as one minus none. For a uniformly chosen hand, count hands containing no target object and subtract from the total. This avoids adding many disjoint cases, such as exactly one, exactly two, and so on. The complement must be taken within the same sample space and under the same selection mechanism.

Arrangement probabilities use equally likely permutations. To count arrangements where two specified objects are adjacent, temporarily treat them as a block and multiply by the internal orders of the block. To require separation, gaps or complement counting may be more convenient. If objects repeat, first determine whether the experiment makes distinct labelled arrangements or distinct visible arrangements equally likely.

A conditional restriction can sometimes be incorporated by changing the denominator to the admissible outcomes, but only after it is clear what information has been given. “Choose uniformly among arrangements with property B” and “make a uniform arrangement, then learn that B occurred” produce the same conditional distribution in a finite uniform model. Other reporting mechanisms may not.

Independent verification can compare ordered and unordered counts, enumerate a small sample space, or sum probabilities over all feasible compositions to check that the total is one. These checks test the modelling as well as the arithmetic.

05 / UNDERSTAND & APPLY

5. Conditioning renormalizes the world you are considering

For P(B) > 0, define P(A|B) = P(A∩B)/P(B). We restrict attention to outcomes in B and rescale their total probability to one. The numerator must represent both the target A and the evidence B, while the denominator represents all outcomes compatible with the evidence.

Conditional probability does not require B to occur earlier in time. It can describe information about a simultaneous measurement or a later observation used to infer an earlier source. The vertical bar means “given this information,” not necessarily “caused by.” Probability alone does not establish a causal relationship.

In a finite uniform model, conditioning on an event B leaves the elementary outcomes in B equally likely. Then P(A|B) can be found by counting |A∩B|/|B| directly. If the original elementary outcomes have unequal probabilities, count-based renormalization is invalid; use their probability weights.

Reversing the condition changes the denominator. In general P(A|B) is not P(B|A). The first measures the fraction of B-compatible outcomes also in A; the second measures the fraction of A-compatible outcomes also in B. Bayes’ theorem will relate these two directions using the prior probabilities.

06 / UNDERSTAND & APPLY

6. Multiplication always uses the appropriate conditional rate

Rearranging the conditional definition gives P(A∩B) = P(A)P(B|A) when P(A) > 0, and also P(B)P(A|B) when P(B) > 0. For sequential events, extend the chain: P(A∩B∩C) = P(A)P(B|A)P(C|A∩B), when the needed conditioning probabilities are defined.

Events A and B are independent when P(A∩B) = P(A)P(B). If P(B) > 0, this is equivalent to P(A|B) = P(A). Independence means learning B does not change the probability of A under this model. It is a mathematical property to be given, derived from the experiment, or checked; it should not be assumed because events sound unrelated.

Disjointness is different. If A and B are disjoint with positive probabilities, then P(A∩B) = 0 while P(A)P(B) > 0, so they are not independent. Learning that one occurred makes the other impossible. Independence of A and B implies independence of their complements, but conditioning on a third event can destroy that independence.

For three or more events, pairwise independence is weaker than mutual independence. With two fair coin tosses, let A be first toss H, B second toss H, and C the tosses matching. Each pair is independent, but P(A∩B∩C) = 1/4 while P(A)P(B)P(C) = 1/8. A triple-product shortcut therefore needs the stronger appropriate independence assumption.

07 / UNDERSTAND & APPLY

7. Add complete paths through a partition

A partition H₁,…,Hₖ consists of mutually exclusive events whose union is the whole sample space. If their probabilities are positive, an event E can occur through each disjoint path Hᵢ∩E. Therefore P(E) = ΣP(Hᵢ)P(E|Hᵢ), the law of total probability.

In a tree diagram, the first branches represent the possible source events Hᵢ. The next branches represent E or its complement conditional on that source. Multiply along a path to obtain a joint probability, then add disjoint paths that lead to the same final event. Branch probabilities leaving one node should sum to one.

The total probability is a weighted average of the conditional probabilities, with weights P(Hᵢ). It must lie between the smallest and largest conditional probabilities. An unweighted arithmetic average is correct only when the source probabilities are equal. Larger sources contribute more outcomes to the total.

If the proposed source cases overlap, they are not a partition and simply summing their joint contributions double-counts outcomes. Refine the cases into disjoint pieces or use a suitable inclusion-exclusion argument. Naming the partition explicitly makes this condition easy to inspect.

08 / UNDERSTAND & APPLY

8. Bayes’ theorem reverses the direction of conditioning

Bayes’ theorem follows by writing the same joint probability two ways. For P(E) > 0, P(Hⱼ|E) = P(Hⱼ)P(E|Hⱼ)/P(E). With a partition, substitute the total-probability denominator ΣP(Hᵢ)P(E|Hᵢ). The numerator is the contribution of the desired source to the evidence; the denominator is the contribution of all sources.

The prior P(Hⱼ) describes the source before observing E. The likelihood P(E|Hⱼ) describes how probable the evidence would be under that source. The posterior P(Hⱼ|E) describes the source after the evidence. A large likelihood alone does not guarantee a large posterior if the source was very rare. The prior weight remains part of the calculation.

A frequency table can make the denominator tangible. In the machine example, imagine 1000 selected items: 600 from A and 400 from B. The expected defect counts are 12 and 20, respectively. Among the 32 expected defective items, 20 came from B. This gives the same conditional ratio as the probability calculation without mistaking a defect rate for a source probability.

Always state the evidence precisely. “A selected item is defective” differs from “a source was selected uniformly and then a defective item was deliberately chosen from it.” The latter involves an additional selection mechanism. Bayes’ theorem updates the model you actually specified; it cannot repair an ambiguous experiment by guessing.

09 / UNDERSTAND & APPLY

9. More observations require the correct joint likelihood

If several observations arise from one chosen source, their joint likelihood may or may not factor into identical independent probabilities. Drawing without replacement from an urn changes its composition, so the probability of two red draws is not generally the square of its original red fraction. Compute the conditional second-draw probability or use combinations.

Bayes can be applied once using the joint evidence, or sequentially by updating after the first observation and then updating again with the correct next-observation likelihood. Both routes agree when the model is consistent. Reusing the original prior after the first observation or assuming independence without replacement breaks that agreement.

An impossible evidence path contributes zero to the denominator. It does not make the whole posterior undefined unless the total evidence probability is zero. If every source gives likelihood zero, the observation is incompatible with the stated model and the elementary conditional formula cannot be applied.

A useful diagram draws each source first and then the two color draws as conditional branches. Mark the changed urn count on the second branch. This makes the dependence explicit and prevents a visually identical pair of “red” labels from being mistaken for two identical probabilities.

10 / UNDERSTAND & APPLY

10. Original practice: events and counting

Use a consistent equally likely sample space in each count. Where only event probabilities are supplied, use the probability identities directly instead of inventing a count model.

11 / UNDERSTAND & APPLY

11. Original practice: name the denominator

For each conditional question, say what the evidence event includes. For each Bayes question, list the prior-weighted evidence paths before dividing.

12 / UNDERSTAND & APPLY

12. A probability audit is a modelling audit

Check that every probability lies in [0,1], all branches from a tree node sum to one, disjoint source paths cover the evidence event, and the posterior probabilities across the source partition sum to one. For total probability, the weighted average should lie between the smallest and largest likelihoods. These checks often reveal a missing path or an incorrect denominator.

The recurring mistakes are treating nonuniform outcome categories as equally likely, mixing ordered and unordered counts, assuming replacement when none is given, confusing independent with disjoint, reversing a conditional without the prior weights, and updating on an imprecisely described report. Each mistake changes the model rather than merely the arithmetic.

A concise complete solution names the sample space or partition, defines the target and evidence events, computes the relevant joint probability, and normalizes by the correct total. For Bayes, the numerator should visibly be one term of the denominator’s sum. This makes the result transparent to someone checking the work.

Finally, interpret the answer in the direction asked. A defect probability given a source is not a source probability given a defect. A posterior can be higher than its prior without being close to one. Evidence changes probabilities through likelihood ratios and base rates, not through the emotional strength of the wording.