Correlation and Regression

Correlation asks whether two variables move together and how closely; regression asks how much one changes when the other does, so that one can be estimated from the other. This chapter covers the scatter diagram, Karl Pearson's coefficient, the two regression lines and their coefficients, Spearman's rank correlation, association of attributes, and — for three variables only, as the syllabus says — partial and multiple correlation and the multiple regression plane.

Scatter diagram and Karl Pearson's r

Plot each pair (x, y) as a point: a cloud rising to the right shows positive correlation, falling to the right negative, and a shapeless cloud no linear correlation. Karl Pearson's coefficient puts a number on it: r = Cov(x, y) ÷ (σₓσᵧ) = Σ(x − x̄)(y − ȳ) ÷ √[Σ(x − x̄)² · Σ(y − ȳ)²].

Worked: five pairs
x	y	x − x̄	y − ȳ	product
1	2	-2	-2	4
2	4	-1	0	0
3	5	0	1	0
4	4	1	0	0
5	5	2	1	2

x̄ = 3, ȳ = 4, Σ(x − x̄)(y − ȳ) = 6, Σ(x − x̄)² = 10, Σ(y − ȳ)² = 6, so r = 6/√60 ≈ 0.77 — a strong positive correlation. Properties: −1 ≤ r ≤ 1; r is a pure number; it is independent of change of origin and scale (if both scale factors have the same sign); and r = 0 means no linear relation, not no relation at all — y = x² over symmetric x gives r = 0.

The two regression lines

There are two regression lines because the question 'estimate y from x' minimises vertical errors and 'estimate x from y' minimises horizontal ones. y on x: y − ȳ = bᵧₓ(x − x̄), with bᵧₓ = r·σᵧ/σₓ. x on y: x − x̄ = bₓᵧ(y − ȳ), with bₓᵧ = r·σₓ/σᵧ.

Both lines pass through the point of means (x̄, ȳ), so solving their equations together gives the means.
r² = bᵧₓ × bₓᵧ, and r takes the common sign of the two coefficients (they always share a sign).
Both coefficients cannot exceed 1 in size at once, because their product r² is at most 1.
Regression coefficients change with scale but not with origin.
If r = ±1 the two lines coincide; if r = 0 they are perpendicular (y = ȳ and x = x̄).

In the worked data bᵧₓ = 6/10 = 0.6 and bₓᵧ = 6/6 = 1; their product 0.6 = r², confirming r ≈ 0.77. The line of y on x is y − 4 = 0.6(x − 3), so at x = 6 the estimate is 5.8.

⚠️ Exam trap

Given two regression equations, you must decide which is y on x. Write each as y = … and x = … ; the pair whose coefficients multiply to at most 1 is the correct assignment. The wrong assignment gives r² > 1, which is impossible.

Rank correlation and association of attributes

When only ranks are available (beauty contest judges, preferences), use Spearman's rank correlation: ρ = 1 − 6Σd² ÷ [n(n² − 1)], where d is the difference between the two ranks of each item. If ranks are tied, give each tied item the average rank and add m(m² − 1)/12 to Σd² for every group of m ties. ρ also lies between −1 and +1.

Two attributes (qualities that are present or absent, such as vaccinated/not and infected/not) are studied with a 2 × 2 table of class frequencies (AB), (Aβ), (αB), (αβ). Yule's coefficient of association is Q = [(AB)(αβ) − (Aβ)(αB)] ÷ [(AB)(αβ) + (Aβ)(αB)], between −1 and +1; Q = 0 means independence. Worked: (AB) = 40, (Aβ) = 10, (αB) = 20, (αβ) = 30 give Q = (1200 − 200)/(1200 + 200) = 5/7 ≈ 0.71, a strong positive association. Attributes are independent when (AB) = (A)(B)/N.

Partial and multiple correlation (three variables)

With three variables X₁, X₂, X₃, the partial correlation r₁₂.₃ measures the link between X₁ and X₂ after removing the influence of X₃: r₁₂.₃ = (r₁₂ − r₁₃r₂₃) ÷ √[(1 − r₁₃²)(1 − r₂₃²)]. The multiple correlation R₁.₂₃ measures how well X₂ and X₃ together predict X₁: R²₁.₂₃ = (r₁₂² + r₁₃² − 2r₁₂r₁₃r₂₃) ÷ (1 − r₂₃²). R lies between 0 and 1 and is never smaller than the size of any simple correlation of X₁ with the other two.

Worked: r₁₂ = 0.7, r₁₃ = r₂₃ = 0.5 give r₁₂.₃ = (0.7 − 0.25)/0.75 = 0.6 and R²₁.₂₃ = (0.49 + 0.25 − 0.35)/0.75 = 0.52, so R₁.₂₃ ≈ 0.72. The multiple regression plane X₁ = a + b₁₂.₃X₂ + b₁₃.₂X₃ is fitted by least squares; b₁₂.₃ is the change in X₁ for a unit change in X₂ with X₃ held fixed.

ℹ️ Note

Correlation is not causation: two series can both rise with time — ice-cream sales and drowning cases both rise in summer — without either causing the other. Such a relation is called spurious or nonsense correlation.

Exam traps and a quick revision

r = Cov/(σₓσᵧ); −1 ≤ r ≤ 1; unchanged by origin and scale.
bᵧₓ = rσᵧ/σₓ, bₓᵧ = rσₓ/σᵧ, r = ±√(bᵧₓ·bₓᵧ) with their common sign.
Regression lines meet at (x̄, ȳ).
Spearman: ρ = 1 − 6Σd²/[n(n² − 1)].
Yule's Q = (ad − bc)/(ad + bc) for the 2 × 2 table a, b, c, d.

Key takeaways

Pearson's r measures linear association and is free of origin and scale.
Two regression lines, both through (x̄, ȳ); r² = bᵧₓ·bₓᵧ.
Spearman's ρ = 1 − 6Σd²/[n(n² − 1)] works from ranks, with a correction for ties.
Yule's Q measures association of attributes; Q = 0 means independence.
r₁₂.₃ removes X₃'s influence; R₁.₂₃ measures joint prediction and lies between 0 and 1.

Practice questions (10)

Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.

Karl Pearson's coefficient of correlation always lies between:
1. −∞ and +∞
2. 0 and ∞
3. 0 and 1
4. −1 and +1
Show answer
Answer: D — −1 and +1
By the Cauchy–Schwarz inequality |Cov(x, y)| ≤ σₓσᵧ, so r lies in [−1, 1], reaching ±1 only for an exact straight-line relation.
The two regression coefficients are bᵧₓ = 0.8 and bₓᵧ = 0.2. The coefficient of correlation is:
1. 0.5
2. 0.16
3. 0.4
4. −0.4
Show answer
Answer: C — 0.4
r² = 0.8 × 0.2 = 0.16, so r = 0.4, positive because both coefficients are positive. 0.16 is r², not r.
If bᵧₓ = −0.9 and bₓᵧ = −0.4, then r equals:
1. 0.36
2. −0.36
3. 0.6
4. −0.6
Show answer
Answer: D — −0.6
r² = (−0.9)(−0.4) = 0.36, so |r| = 0.6, and r takes the common negative sign of the coefficients: r = −0.6.
The two regression lines intersect at the point:
1. (σₓ, σᵧ)
2. (0, 0)
3. (x̄, ȳ)
4. (bₓᵧ, bᵧₓ)
Show answer
Answer: C — (x̄, ȳ)
Each line has the form (y − ȳ) = b(x − x̄) or (x − x̄) = b(y − ȳ), and both are satisfied by x = x̄, y = ȳ. That is why solving the two equations gives the means.
The regression lines of a bivariate data set are x + 2y = 5 and 2x + 3y = 8. The means x̄ and ȳ are:
1. x̄ = 1, ȳ = 2
2. x̄ = 5, ȳ = 0
3. x̄ = 4, ȳ = 0
4. x̄ = 2, ȳ = 1
Show answer
Answer: A — x̄ = 1, ȳ = 2
Both lines pass through (x̄, ȳ). From the first, x = 5 − 2y; substituting, 10 − 4y + 3y = 8, so y = 2 and x = 1.
If r = 0.5, σₓ = 4 and σᵧ = 6, the regression coefficient of y on x is:
1. 3
2. 0.33
3. 1.5
4. 0.75
Show answer
Answer: D — 0.75
bᵧₓ = r·σᵧ/σₓ = 0.5 × 6/4 = 0.75. Swapping the SDs gives bₓᵧ = 0.5 × 4/6 ≈ 0.33.
For 5 pairs of ranks the sum of squared rank differences is Σd² = 6. Spearman's rank correlation is:
1. 0.3
2. 0.94
3. 0.7
4. 0.6
Show answer
Answer: C — 0.7
ρ = 1 − 6 × 6/[5(25 − 1)] = 1 − 36/120 = 1 − 0.3 = 0.7. Stopping at 36/120 gives the wrong 0.3.
If each x is replaced by (x − 10)/2 and each y by (y − 50)/5, the correlation coefficient:
1. is multiplied by 2.5
2. is divided by 10
3. changes sign
4. remains unchanged
Show answer
Answer: D — remains unchanged
r is independent of change of origin and scale when the scale factors have the same sign; here both divisors (2 and 5) are positive.
In a 2 × 2 table of two attributes, (AB) = 40, (Aβ) = 10, (αB) = 20 and (αβ) = 30. Yule's coefficient of association Q is:
1. 6/7
2. 1/2
3. 2/7
4. 5/7
Show answer
Answer: D — 5/7
Q = [(AB)(αβ) − (Aβ)(αB)]/[(AB)(αβ) + (Aβ)(αB)] = (1200 − 200)/(1200 + 200) = 1000/1400 = 5/7 ≈ 0.71, a strong positive association.
Given r₁₂ = 0.7, r₁₃ = 0.5 and r₂₃ = 0.5, the partial correlation coefficient r₁₂.₃ is:
1. 0.6
2. 0.7
3. 0.45
4. 0.52
Show answer
Answer: A — 0.6
r₁₂.₃ = (r₁₂ − r₁₃r₂₃)/√[(1 − r₁₃²)(1 − r₂₃²)] = (0.7 − 0.25)/√(0.75 × 0.75) = 0.45/0.75 = 0.6. 0.52 is R²₁.₂₃ for the same data.