Understanding Correlation in Python: Pearson vs. Spearman Explained
Understanding Correlation in Python: Pearson vs. Spearman Explained
When exploring relationships between variables in data science, correlation is often the first metric we reach for. Whether you are analyzing student study habits, financial trends, or marketing conversions, quantifying how variables move together gives immediate insight into your data.
However, a common question arises: Which correlation metric should you use—Pearson or Spearman?
In this post, we’ll break down the conceptual differences between Pearson and Spearman correlation, walk through a practical Python example, and visualize the results using
matplotlib.1. What Is Correlation?
Correlation measures the statistical relationship between two continuous variables:
- Pearson Correlation ($r$): Measures the strength and direction of the linear relationship between two variables. It assumes your data is normally distributed and continuous.
- Values range from -1.0 (perfect negative linear relationship) to +1.0 (perfect positive linear relationship).
- A value of 0 indicates no linear relationship.
- Spearman Rank Correlation ($\rho$ or $r_s$): Measures the strength and direction of a monotonic (rank-based) relationship.
- Instead of evaluating raw values, it converts values to ranks before computing correlation.
- As long as variable $Y$ consistently increases whenever $X$ increases—even if the increase isn't strictly linear—Spearman’s coefficient will be $+1.0$.
- It is far less sensitive to outliers and does not assume a normal distribution.
2. A Real-World Example: Study Hours vs. Marks
Let’s test both metrics on a classic scenario: How do study hours correlate with exam marks?
Prerequisites
You'll need
numpy, scipy, and matplotlib:Bash
pip install numpy scipy matplotlib
The Code
Python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
# 1. Prepare sample data
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
marks = np.array([45, 50, 54, 61, 65, 72, 78, 84])
# 2. Calculate Pearson and Spearman correlation
pearson = stats.pearsonr(hours, marks)
spearman = stats.spearmanr(hours, marks)
print("Pearson r :", round(pearson.statistic, 4))
print("Spearman ฯ :", round(spearman.statistic, 4))
# 3. Visualize the relationship with a fitted regression line
plt.figure(figsize=(8, 4.5))
# Scatter plot of raw data points
plt.scatter(hours, marks, s=70, color="#1f77b4", label="Observed Marks")
# Calculate and plot linear trend line (degree 1 polynomial)
trendline = np.polyval(np.polyfit(hours, marks, 1), hours)
plt.plot(hours, trendline, linewidth=2, color="#1f77b4", label="Trendline")
# Plot styling
plt.title("Study Hours vs Marks", fontsize=15, weight="bold")
plt.xlabel("Study Hours", fontsize=12)
plt.ylabel("Marks", fontsize=12)
plt.grid(True, linestyle="--", alpha=0.5)
plt.tight_layout()
plt.show()
3. Interpreting the Output
When you run the code, you get the following results:
Plaintext
Pearson r : 0.998
Spearman ฯ : 1.0
Why are the two values slightly different?
- Spearman $\rho = 1.0$: Every time study hours increase, the marks also strictly increase ($45 \to 50 \to 54 \to 61 \dots$). Because the rank ordering is strictly preserved with zero inversions, the monotonic relationship is perfect.
- Pearson $r = 0.998$: The relationship is nearly a straight line, but the mark increments per hour vary slightly ($+5$, $+4$, $+7$, $+4$, $+7$, $+6$, $+6$). Because it is not an exact mathematical line ($y = mx + c$), the linear score is just shy of 1.0 (though still an extremely strong positive correlation).
4. Cheat Sheet: When to Use Which?
| Scenario | Preferred Metric | Reason |
| Data is normally distributed and strictly linear | Pearson | Maximizes statistical power on interval/ratio data. |
| Data has outliers or skewness | Spearman | Ranking values dampens the extreme influence of outliers. |
| Monotonic, non-linear trend (e.g., exponential growth) | Spearman | Captures whether $Y$ grows with $X$ regardless of curvature. |
| Ordinal data (e.g., Likert survey ratings 1–5) | Spearman | Natural fit for ranked categories. |
Summary
- Pearson checks: "Do these data points form a straight line?"
- Spearman checks: "When one value goes up, does the other value consistently go up too?"
Using Python's
scipy.stats makes calculating both effortless. Pairing statistical calculations with visualizations (like scatter plots and trendlines) gives you the full picture before building machine learning models or making business decisions.
Comments
Post a Comment