Probability Theory

Probability is the language of uncertainty, and every later chapter — distributions, sampling, inference — is written in it. This chapter gives the meaning of probability and its classical, relative-frequency and axiomatic definitions, the addition and multiplication theorems, conditional and compound probability, the difference between independent and mutually exclusive events, the theorem of total probability and Bayes' theorem.

What probability means: three definitions

An experiment with uncertain outcome has a sample space S of all possible outcomes; an event is a subset of S. Probability assigns each event a number from 0 (impossible) to 1 (certain). There are three standard ways to define that number, and each has a weakness the next one fixes.

The three definitions
DefinitionStatementLimitation
Classical (a priori)P(A) = favourable cases ÷ total cases, outcomes equally likely and finitefails if outcomes are not equally likely or infinite
Relative frequency (empirical)P(A) = limit of m/n as the number of trials n growsneeds repeatable trials; the limit is never observed exactly
Axiomatic (Kolmogorov)P(A) ≥ 0; P(S) = 1; P(A ∪ B) = P(A) + P(B) for disjoint A, Bsays how probabilities behave, not how to find them

A fourth, subjective probability, is a degree of personal belief — used for one-off events such as a new product's success. The odds in favour of A are P(A) : P(A′).

Addition and multiplication theorems

Addition theorem: P(A ∪ B) = P(A) + P(B) − P(A ∩ B); for mutually exclusive events the last term is zero. For three events, P(A ∪ B ∪ C) = ΣP(A) − ΣP(A ∩ B) + P(A ∩ B ∩ C). Conditional probability of A given B is P(A | B) = P(A ∩ B) ÷ P(B), P(B) > 0. Rearranged, it is the multiplication theorem for compound events: P(A ∩ B) = P(B)·P(A | B) = P(A)·P(B | A).

Worked: drawing two cards one after the other without replacement, P(both aces) = (4/52)(3/51) = 1/221. With replacement it would be (4/52)² = 1/169, because the second draw no longer depends on the first.

Independent against mutually exclusive

Mutually exclusive events cannot happen together: P(A ∩ B) = 0. Independent events do not affect each other: P(A ∩ B) = P(A)·P(B), equivalently P(A | B) = P(A). These are different ideas, and in fact two events with positive probabilities that are mutually exclusive are never independent — knowing that A happened tells you B did not.

  • If A and B are independent, so are A and B′, A′ and B, and A′ and B′.
  • P(at least one of independent A, B) = 1 − P(A′)P(B′).
  • Two throws of a die: P(at least one six) = 1 − (5/6)² = 11/36.
⚠️ Exam trap
'At least one' questions are fastest by the complement. Adding P(six on first) + P(six on second) = 2/6 double-counts the case of two sixes.

Total probability and Bayes' theorem

If B₁, B₂, …, Bₙ are mutually exclusive and together make up the whole sample space, then for any event A, P(A) = Σ P(Bᵢ)P(A | Bᵢ) (total probability). Bayes' theorem turns the conditioning around: P(Bᵢ | A) = P(Bᵢ)P(A | Bᵢ) ÷ Σ P(Bⱼ)P(A | Bⱼ). The P(Bᵢ) are prior probabilities, the P(Bᵢ | A) posterior ones.

Worked Bayes: which machine made the defective item?
MachineShare of outputDefect rateProductPosterior
A0.600.020.0120.5
B0.400.030.0120.5
Total1—0.0241

A defective item is equally likely to have come from either machine, although A makes more of the output — B's higher defect rate exactly balances A's larger share.

Exam traps and a quick revision

  • 0 ≤ P(A) ≤ 1 and P(A′) = 1 − P(A).
  • P(A ∪ B) = P(A) + P(B) − P(A ∩ B).
  • P(A | B) = P(A ∩ B)/P(B); independent means P(A ∩ B) = P(A)P(B).
  • Mutually exclusive events with positive probabilities are dependent.
  • Bayes: posterior ∝ prior × likelihood, normalised by the total probability.

Key takeaways

  • Classical probability needs equally likely outcomes; relative frequency needs repeatable trials; the axiomatic approach states the rules.
  • Addition: P(A ∪ B) = P(A) + P(B) − P(A ∩ B); multiplication: P(A ∩ B) = P(A)P(B | A).
  • Independence and mutual exclusivity are different; with positive probabilities they cannot hold together.
  • Total probability: P(A) = ΣP(Bᵢ)P(A | Bᵢ) over a partition B₁ … Bₙ.
  • Bayes: P(Bᵢ | A) = P(Bᵢ)P(A | Bᵢ)/ΣP(Bⱼ)P(A | Bⱼ).

Practice questions (10)

Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.

  1. The classical definition of probability requires that the outcomes of the experiment be:

    1. based on past records
    2. personal beliefs
    3. infinite and independent
    4. finite in number and equally likely
    Show answer

    Answer: D — finite in number and equally likely

    P(A) = favourable ÷ total only makes sense when the cases can be counted and each is equally likely. Past records give the relative-frequency definition; beliefs give subjective probability.

  2. The axiomatic definition of probability was given by:

    1. A. N. Kolmogorov
    2. R. A. Fisher
    3. Thomas Bayes
    4. Karl Pearson
    Show answer

    Answer: A — A. N. Kolmogorov

    Kolmogorov set out the axioms in 1933: non-negativity, P(S) = 1 and additivity over disjoint events. Bayes is known for the theorem on inverse probability.

  3. If P(A) = 0.5, P(B) = 0.4 and P(A ∩ B) = 0.2, then P(A ∪ B) is:

    1. 1.1
    2. 0.7
    3. 0.9
    4. 0.2
    Show answer

    Answer: B — 0.7

    P(A ∪ B) = 0.5 + 0.4 − 0.2 = 0.7. Adding without subtracting the overlap gives 0.9, which counts A ∩ B twice.

  4. With P(A) = 0.5, P(B) = 0.4 and P(A ∩ B) = 0.2, the conditional probability P(A | B) is:

    1. 0.4
    2. 0.8
    3. 0.2
    4. 0.5
    Show answer

    Answer: D — 0.5

    P(A | B) = P(A ∩ B)/P(B) = 0.2/0.4 = 0.5. Here P(A | B) = P(A), so A and B are in fact independent.

  5. A and B are independent with P(A) = 0.3 and P(B) = 0.4. P(A ∪ B) is:

    1. 0.7
    2. 0.58
    3. 0.12
    4. 0.42
    Show answer

    Answer: B — 0.58

    P(A ∩ B) = 0.3 × 0.4 = 0.12, so P(A ∪ B) = 0.3 + 0.4 − 0.12 = 0.58. Equivalently 1 − (0.7)(0.6) = 0.58.

  6. Two events A and B have P(A) > 0 and P(B) > 0 and are mutually exclusive. Then A and B:

    1. have P(A ∪ B) = 0
    2. must be independent
    3. are exhaustive
    4. cannot be independent
    Show answer

    Answer: D — cannot be independent

    Mutual exclusivity gives P(A ∩ B) = 0, while independence would need P(A ∩ B) = P(A)P(B) > 0. Both cannot hold, so the events are dependent.

  7. Two cards are drawn one after the other without replacement from a pack of 52. The probability that both are aces is:

    1. 1/221
    2. 1/26
    3. 2/13
    4. 1/169
    Show answer

    Answer: A — 1/221

    P = (4/52) × (3/51) = 12/2652 = 1/221. 1/169 = (4/52)² is the answer with replacement.

  8. A fair die is thrown twice. The probability of getting at least one six is:

    1. 1/3
    2. 25/36
    3. 11/36
    4. 1/36
    Show answer

    Answer: C — 11/36

    P(no six) = (5/6)² = 25/36, so P(at least one) = 11/36. 1/3 = 2/6 double-counts the double six.

  9. Machine A makes 60% of a factory's output with 2% defective; machine B makes 40% with 3% defective. An item chosen at random is defective. The probability that it came from A is:

    1. 2/5
    2. 3/5
    3. 0.024
    4. 1/2
    Show answer

    Answer: D — 1/2

    By Bayes: P(A | D) = (0.6 × 0.02)/(0.6 × 0.02 + 0.4 × 0.03) = 0.012/0.024 = 1/2. 0.024 is the total probability of a defective, and 3/5 is only the prior share of A.

  10. For two events, P(A) = 0.6, P(B) = 0.5 and P(A ∪ B) = 0.8. The probability that exactly one of them occurs is:

    1. 0.3
    2. 0.8
    3. 0.5
    4. 1.1
    Show answer

    Answer: C — 0.5

    P(A ∩ B) = 0.6 + 0.5 − 0.8 = 0.3, and exactly one = P(A ∪ B) − P(A ∩ B) = 0.8 − 0.3 = 0.5.