Understanding Correlation in Python: Pearson vs. Spearman Explained


Understanding Correlation in Python: Pearson vs. Spearman Explained

When exploring relationships between variables in data science, correlation is often the first metric we reach for. Whether you are analyzing student study habits, financial trends, or marketing conversions, quantifying how variables move together gives immediate insight into your data.

However, a common question arises: Which correlation metric should you use—Pearson or Spearman?

In this post, we’ll break down the conceptual differences between Pearson and Spearman correlation, walk through a practical Python example, and visualize the results using matplotlib.

1. What Is Correlation?

Correlation measures the statistical relationship between two continuous variables:

  • Pearson Correlation ($r$): Measures the strength and direction of the linear relationship between two variables. It assumes your data is normally distributed and continuous.

    • Values range from -1.0 (perfect negative linear relationship) to +1.0 (perfect positive linear relationship).

    • A value of 0 indicates no linear relationship.

  • Spearman Rank Correlation ($\rho$ or $r_s$): Measures the strength and direction of a monotonic (rank-based) relationship.

    • Instead of evaluating raw values, it converts values to ranks before computing correlation.

    • As long as variable $Y$ consistently increases whenever $X$ increases—even if the increase isn't strictly linear—Spearman’s coefficient will be $+1.0$.

    • It is far less sensitive to outliers and does not assume a normal distribution.

2. A Real-World Example: Study Hours vs. Marks

Let’s test both metrics on a classic scenario: How do study hours correlate with exam marks?

Prerequisites

You'll need numpy, scipy, and matplotlib:

Bash
pip install numpy scipy matplotlib

The Code

Python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# 1. Prepare sample data
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
marks = np.array([45, 50, 54, 61, 65, 72, 78, 84])

# 2. Calculate Pearson and Spearman correlation
pearson = stats.pearsonr(hours, marks)
spearman = stats.spearmanr(hours, marks)

print("Pearson r  :", round(pearson.statistic, 4))
print("Spearman ฯ :", round(spearman.statistic, 4))

# 3. Visualize the relationship with a fitted regression line
plt.figure(figsize=(8, 4.5))

# Scatter plot of raw data points
plt.scatter(hours, marks, s=70, color="#1f77b4", label="Observed Marks")

# Calculate and plot linear trend line (degree 1 polynomial)
trendline = np.polyval(np.polyfit(hours, marks, 1), hours)
plt.plot(hours, trendline, linewidth=2, color="#1f77b4", label="Trendline")

# Plot styling
plt.title("Study Hours vs Marks", fontsize=15, weight="bold")
plt.xlabel("Study Hours", fontsize=12)
plt.ylabel("Marks", fontsize=12)
plt.grid(True, linestyle="--", alpha=0.5)
plt.tight_layout()
plt.show()

3. Interpreting the Output

When you run the code, you get the following results:

Plaintext
Pearson r  : 0.998
Spearman ฯ : 1.0

Why are the two values slightly different?

  • Spearman $\rho = 1.0$: Every time study hours increase, the marks also strictly increase ($45 \to 50 \to 54 \to 61 \dots$). Because the rank ordering is strictly preserved with zero inversions, the monotonic relationship is perfect.

  • Pearson $r = 0.998$: The relationship is nearly a straight line, but the mark increments per hour vary slightly ($+5$, $+4$, $+7$, $+4$, $+7$, $+6$, $+6$). Because it is not an exact mathematical line ($y = mx + c$), the linear score is just shy of 1.0 (though still an extremely strong positive correlation).

4. Cheat Sheet: When to Use Which?

ScenarioPreferred MetricReason
Data is normally distributed and strictly linearPearsonMaximizes statistical power on interval/ratio data.
Data has outliers or skewnessSpearmanRanking values dampens the extreme influence of outliers.
Monotonic, non-linear trend (e.g., exponential growth)SpearmanCaptures whether $Y$ grows with $X$ regardless of curvature.
Ordinal data (e.g., Likert survey ratings 1–5)SpearmanNatural fit for ranked categories.

Summary

  • Pearson checks: "Do these data points form a straight line?"


  • Spearman checks: "When one value goes up, does the other value consistently go up too?"


Using Python's scipy.stats makes calculating both effortless. Pairing statistical calculations with visualizations (like scatter plots and trendlines) gives you the full picture before building machine learning models or making business decisions.

Comments

Popular posts from this blog

I am from mechanical background.. how I can start with python? Tell me career opportunities also .

Root Finding in Python: Solving and Visualizing Equations with SciPy

Here's a step-by-step guide to help you install Visual Studio Code (VSCode) for Python on Windows