Pearson vs. Spearman Correlation in Python: Understanding the Difference
Pearson vs. Spearman Correlation in Python: Understanding the Difference
When analyzing relationships between two continuous variables, correlation is usually the first statistical tool we reach for. However, picking the right metric matters. In data science, the two most common metrics are Pearson correlation and Spearman rank correlation.
Here is a practical breakdown of how they work, how they differ, and how to compute both using Python.
The Core Concept
- Pearson Correlation ($r$): Measures the strength and direction of a linear relationship between two continuous variables. It evaluates whether the data points fall directly on a straight line.
- Spearman Correlation ($\rho$): Measures the strength and direction of a monotonic relationship based on data rankings. It assesses whether one variable increases (or decreases) as the other does, regardless of whether that increase is strictly linear.
Practical Example: Study Hours vs. Marks
Consider a dataset tracking the number of hours studied alongside the marks obtained on an exam:
Python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
# Data
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
marks = np.array([45, 50, 54, 61, 65, 72, 78, 84])
# Calculate correlations
pearson = stats.pearsonr(hours, marks)
spearman = stats.spearmanr(hours, marks)
print("Pearson r :", round(pearson.statistic, 4))
print("Spearman ฯ:", round(spearman.statistic, 4))
Output:
Plaintext
Pearson r : 0.998
Spearman ฯ: 1.0
Visualizing the Fit
Python
plt.figure(figsize=(8, 4.5))
plt.scatter(hours, marks, s=70, color='#1f77b4', label='Observations')
# Fit linear regression trend line
plt.plot(hours, np.polyval(np.polyfit(hours, marks, 1), hours), linewidth=2)
plt.title("Study Hours vs Marks", fontsize=15, weight="bold")
plt.xlabel("Study Hours")
plt.ylabel("Marks")
plt.tight_layout()
plt.show()
Why the Results Differ
- Spearman is exactly $1.0$: Every time
hoursincreases,marksstrictly increases. Because the relative order (rank) is perfectly preserved, the rank correlation is a perfect $+1.0$. - Pearson is slightly less than $1.0$ ($0.998$): While marks consistently increase, the step increments are not perfectly uniform (e.g., $+5$, $+4$, $+7$, $+4$, $+7$, $+6$, $+6$). Because the points deviate ever so slightly from a mathematically perfect straight line, $r$ is just under $1.0$.
When Should You Use Which?
| Feature | Pearson Correlation | Spearman Correlation |
| Relationship Type | Linear only | Monotonic (linear or non-linear) |
| Data Types | Continuous, normally distributed | Continuous or Ordinal |
| Sensitivity to Outliers | High (sensitive to extreme values) | Low (robust to outliers) |
| Underlying Math | Raw values and variances | Ranks of values |
- Use Pearson when your data is normally distributed, free of extreme outliers, and you specifically care about a straight-line trend.
- Use Spearman when your data is skewed, contains outliers, or when your relationship is monotonic but curved (exponential, logarithmic, or rank-based).
Comments
Post a Comment