Correlation and Regression (Extended)

This document covers scatter diagrams, the product moment correlation coefficient, Spearman’s rank Correlation, least squares regression, and residual analysis.

A scatter diagram (scatter plot) displays pairs of values $(x_i, y_i)$ as points on a Coordinate grid. Visual inspection reveals:

• The direction of association (positive, negative, or none).
• The strength of association (strong, moderate, weak).
• The shape of the relationship (linear, curved, clustered).
• The presence of outliers.

An outlier is a data point that lies far from the general pattern. Outliers can:

• Be genuine extreme values.
• Result from measurement errors.
• Significantly affect the correlation coefficient and regression line.

The product moment correlation coefficient (also called Pearson’s correlation coefficient) For a sample of $n$ pairs $(x_i, y_i)$ is:

$r = \frac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}$

Where:

$S_{xy} = \sum(x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - n\bar{x}\bar{y}$

$S_{xx} = \sum(x_i - \bar{x})^2 = \sum x_i^2 - n\bar{x}^2$

$S_{yy} = \sum(y_i - \bar{y})^2 = \sum y_i^2 - n\bar{y}^2$

• $-1 \leq r \leq 1$.
• $r = 1$: perfect positive linear correlation.
• $r = -1$: perfect negative linear correlation.
• $r = 0$: no linear correlation (but there may be a non-linear relationship).
• $r$ is independent of the units of measurement.
• $r$ is unchanged if both variables are transformed linearly ($x' = ax + b$, $y' = cy + d$ with $a, c \gt 0$).

Proof. By the Cauchy-Schwarz inequality:

$\left(\sum a_i b_i\right)^2 \leq \left(\sum a_i^2\right)\!\left(\sum b_i^2\right)$

Setting $a_i = x_i - \bar{x}$ and $b_i = y_i - \bar{y}$:

$S_{xy}^2 \leq S_{xx}\,S_{yy}$

$r^2 = \frac{S_{xy}^2}{S_{xx}\,S_{yy}} \leq 1 \implies |r| \leq 1 \quad \blacksquare$

Problem. Find the PMCC for the following data:

n = 5$$\bar{x} = 6$$\bar{y} = 5.6.

$\sum x_i y_i = 6 + 20 + 24 + 56 + 90 = 196$

$S_{xy} = 196 - 5(6)(5.6) = 196 - 168 = 28$

\sum x_i^2 = 4 + 16 + 36 + 64 + 100 = 220$$S_{xx} = 220 - 5(36) = 40

\sum y_i^2 = 9 + 25 + 16 + 49 + 81 = 180$$S_{yy} = 180 - 5(31.36) = 180 - 156.8 = 23.2

$r = \frac{28}{\sqrt{40 \times 23.2}} = \frac{28}{\sqrt{928}} = \frac{28}{30.46} \approx 0.919$

This indicates strong positive linear correlation.

When data values are large, coding simplifies calculations. Use $u = \dfrac{x - a}{c}$ and $v = \dfrac{y - b}{d}$ where $a, b$ are shift values and $c, d$ are scaling values.

The PMCC is unchanged by coding: $r_{xy} = r_{uv}$.

Spearman’s rank correlation coefficient $r_s$ measures the strength of the monotonic Relationship between two variables:

$r_s = 1 - \frac{6\sum d_i^2}{n(n^2 - 1)}$

Where $d_i = \mathrm{rank}(x_i) - \mathrm{rank}(y_i)$ is the difference in ranks for the $i$-th pair.

• Data is ordinal (ranked categories).
• The relationship is monotonic but not necessarily linear.
• There are significant outliers that would distort the PMCC.
• The data contains tied ranks.

When values are tied, assign the average rank to all tied values. For example, if two values Are tied for ranks 3 and 4, both receive rank 3.5.

When ties exist, the simplified formula is only approximate. A more accurate formula uses:

$r_s = \frac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}$

Applied to the rank data.

Problem. Two judges rank 6 competitors:

$\sum d_i^2 = 1 + 4 + 1 + 1 + 1 + 4 = 12$

$r_s = 1 - \frac{6 \times 12}{6(35)} = 1 - \frac{72}{210} = 1 - 0.343 = 0.657$

This indicates moderate positive agreement between the judges.

Problem. Find $r_s$ for the following data:

Ranks of $x$: 1, 2.5, 2.5, 4, 5 (tied at 20).

Ranks of $y$: 1, 2, 3, 4, 5.

$d_i$: 0, 0.5, -0.5, 0, 0.

$\sum d_i^2 = 0 + 0.25 + 0.25 + 0 + 0 = 0.5$

$r_s = 1 - \frac{6 \times 0.5}{5 \times 24} = 1 - \frac{3}{120} = 1 - 0.025 = 0.975$

Very strong positive monotonic relationship.

The least squares regression line of $y$ on $x$ is the line $y = a + bx$ that minimises the Sum of squared residuals:

$S = \sum_{i=1}^{n}(y_i - a - bx_i)^2$

Setting $\dfrac{\partial S}{\partial a} = 0$ and $\dfrac{\partial S}{\partial b} = 0$:

$b = \frac{S_{xy}}{S_{xx}} = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2}$

$a = \bar{y} - b\bar{x}$

Key property: The regression line always passes through the point $(\bar{x}, \bar{y})$.

$\frac{\partial S}{\partial a} = -2\sum(y_i - a - bx_i) = 0 \implies na + b\sum x_i = \sum y_i$

$\frac{\partial S}{\partial b} = -2\sum x_i(y_i - a - bx_i) = 0 \implies a\sum x_i + b\sum x_i^2 = \sum x_i y_i$

These are the normal equations. Dividing the first by $n$ gives $\bar{y} = a + b\bar{x}$ Confirming the line passes through the mean point.

Using the data from Section 2.4:

$b = \dfrac{S_{xy}}{S_{xx}} = \dfrac{28}{40} = 0.7$

$a = \bar{y} - b\bar{x} = 5.6 - 0.7(6) = 5.6 - 4.2 = 1.4$

$y = 1.4 + 0.7x$

To predict $y$ when $x = 7$: $y = 1.4 + 4.9 = 6.3$.

The regression line of $x$ on $y$ (used when predicting $x$ from $y$) is:

$x = \bar{x} + \frac{S_{xy}}{S_{yy}}(y - \bar{y})$

Important: The two regression lines are different unless $|r| = 1$. The line of $y$ on $x$ Minimises vertical residuals; the line of $x$ on $y$ minimises horizontal residuals.

1. Interpolation (predicting within the data range) is generally reliable.
2. Extrapolation (predicting outside the data range) is unreliable — the relationship may not hold.
3. The regression line assumes a linear relationship.
4. The model assumes the residuals are independent and normally distributed with constant variance (homoscedasticity).

Mathematics provides the language for describing patterns, relationships, and change. Functions transform inputs to outputs like machines, calculus measures how things change and accumulate, and probability quantifies uncertainty. The power of mathematics lies in its ability to model real-world situations abstractly, allowing us to solve problems and make predictions across science, engineering, and everyday life.

• Algebra
• Calculus
• Statistics
• Trigonometry