Pearson vs. Spearman Correlation in Python: Understanding the Difference

 

Pearson vs. Spearman Correlation in Python: Understanding the Difference

When analyzing relationships between two continuous variables, correlation is usually the first statistical tool we reach for. However, picking the right metric matters. In data science, the two most common metrics are Pearson correlation and Spearman rank correlation.

Here is a practical breakdown of how they work, how they differ, and how to compute both using Python.

The Core Concept

  • Pearson Correlation ($r$): Measures the strength and direction of a linear relationship between two continuous variables. It evaluates whether the data points fall directly on a straight line.

  • Spearman Correlation ($\rho$): Measures the strength and direction of a monotonic relationship based on data rankings. It assesses whether one variable increases (or decreases) as the other does, regardless of whether that increase is strictly linear.

Practical Example: Study Hours vs. Marks

Consider a dataset tracking the number of hours studied alongside the marks obtained on an exam:

Python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

# Data
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
marks = np.array([45, 50, 54, 61, 65, 72, 78, 84])

# Calculate correlations
pearson = stats.pearsonr(hours, marks)
spearman = stats.spearmanr(hours, marks)

print("Pearson r :", round(pearson.statistic, 4))
print("Spearman ฯ:", round(spearman.statistic, 4))

Output:

Plaintext
Pearson r : 0.998
Spearman ฯ: 1.0

Visualizing the Fit

Python
plt.figure(figsize=(8, 4.5))
plt.scatter(hours, marks, s=70, color='#1f77b4', label='Observations')

# Fit linear regression trend line
plt.plot(hours, np.polyval(np.polyfit(hours, marks, 1), hours), linewidth=2)

plt.title("Study Hours vs Marks", fontsize=15, weight="bold")
plt.xlabel("Study Hours")
plt.ylabel("Marks")
plt.tight_layout()
plt.show()

Why the Results Differ

  1. Spearman is exactly $1.0$: Every time hours increases, marks strictly increases. Because the relative order (rank) is perfectly preserved, the rank correlation is a perfect $+1.0$.

  2. Pearson is slightly less than $1.0$ ($0.998$): While marks consistently increase, the step increments are not perfectly uniform (e.g., $+5$, $+4$, $+7$, $+4$, $+7$, $+6$, $+6$). Because the points deviate ever so slightly from a mathematically perfect straight line, $r$ is just under $1.0$.

When Should You Use Which?

FeaturePearson CorrelationSpearman Correlation
Relationship TypeLinear onlyMonotonic (linear or non-linear)
Data TypesContinuous, normally distributedContinuous or Ordinal
Sensitivity to OutliersHigh (sensitive to extreme values)Low (robust to outliers)
Underlying MathRaw values and variancesRanks of values
  • Use Pearson when your data is normally distributed, free of extreme outliers, and you specifically care about a straight-line trend.

  • Use Spearman when your data is skewed, contains outliers, or when your relationship is monotonic but curved (exponential, logarithmic, or rank-based).

Comments

Popular posts from this blog

I am from mechanical background.. how I can start with python? Tell me career opportunities also .

Root Finding in Python: Solving and Visualizing Equations with SciPy

Here's a step-by-step guide to help you install Visual Studio Code (VSCode) for Python on Windows