Regression Metrics (MAE, MSE, RMSE, R²)

1. Introduction

Everything covered so far in this module (Accuracy, Precision, Recall, F1, ROC-AUC) applies to CLASSIFICATION problems. But Linear Regression and Multiple Linear Regression (Module 4) predict continuous numbers — so we need a completely different set of metrics to evaluate them. This topic covers the four most important regression evaluation metrics.

2. What are Regression Metrics?

Simple definition: Regression metrics measure how close a regression model's predicted numeric values are to the actual, true numeric values.

Technical explanation: Regression metrics quantify prediction error by comparing predicted values to actual values across a dataset, using different mathematical approaches — averaging absolute differences (MAE), averaging squared differences (MSE), taking the square root of MSE for interpretability (RMSE), or measuring the proportion of variance explained (R²).

3. Why is it Important?

• Classification metrics (Accuracy, Precision, etc.) are meaningless for regression — you need metrics specifically designed for continuous predictions.
• Different metrics emphasize different aspects of error (e.g., MSE penalizes large errors more heavily than MAE).
• R² provides an intuitive, easily communicated measure of "how well does this model explain the data."

4. Prerequisites

Comfort with Linear Regression (Module 4, Topic 1) and Basic Statistics (Module 1, Topic 8).

5. Core Concepts

1. Mean Absolute Error (MAE)
2. Mean Squared Error (MSE)
3. Root Mean Squared Error (RMSE)
4. R² Score (Coefficient of Determination)

6. Detailed Explanation

a) Mean Absolute Error (MAE)

MAE calculates the average of the ABSOLUTE differences between predicted and actual values — treating all errors equally regardless of direction (over- or under-prediction) or size, making it simple and directly interpretable in the original units.

b) Mean Squared Error (MSE)

MSE calculates the average of the SQUARED differences between predicted and actual values (this is the same cost function Linear Regression itself minimizes during training, Module 4, Topic 1). Squaring means larger errors are penalized disproportionately more than smaller ones.

c) Root Mean Squared Error (RMSE)

RMSE is simply the square root of MSE, bringing the error metric back into the same units as the original target variable — making it more directly interpretable than raw MSE, while still penalizing large errors more heavily than MAE.

d) R² Score (Coefficient of Determination)

R² measures the proportion of variance in the target variable that the model successfully explains, ranging (typically) from 0 to 1 — an R² of 1.0 means the model perfectly explains all variance in the data; an R² of 0 means it explains none (performing no better than simply predicting the mean every time).

7. How It Works

1. Train a regression model and generate predictions on a test set.
2. Calculate the difference between each prediction and its corresponding actual value.
3. Depending on the metric: take the average absolute difference (MAE), average squared difference (MSE), square root of that average (RMSE), or compare total variance explained (R²).
4. Interpret the resulting metric(s) in the context of your specific problem.

8. Real-World Example

A house price prediction model with an MAE of $15,000 means, on average, its predictions are off by about $15,000 in either direction — a fairly intuitive, directly interpretable number. Meanwhile, its R² score of 0.85 tells you the model successfully explains 85% of the variation in house prices based on the given features — a different, complementary way of understanding the same model's overall quality.

9. Mathematical Explanation

Mean Absolute Error (MAE):

MAE = (1/n) × Σ|yᵢ − ŷᵢ|

Mean Squared Error (MSE):

MSE = (1/n) × Σ(yᵢ − ŷᵢ)²

Root Mean Squared Error (RMSE):

RMSE = √MSE

R² Score:

R² = 1 − (Σ(yᵢ − ŷᵢ)² / Σ(yᵢ − ȳ)²)

Where:

• yᵢ = actual value for data point i
• ŷᵢ = predicted value for data point i
• ȳ = mean of all actual values
• n = number of data points

Numerical Example:

Actual values: [100, 150, 200], Predicted values: [110, 140, 210]

• Errors: (100-110)=-10, (150-140)=10, (200-210)=-10
• MAE = (|−10|+|10|+|−10|)/3 = (10+10+10)/3 = 10
• MSE = ((-10)²+(10)²+(-10)²)/3 = (100+100+100)/3 = 100
• RMSE = √100 = 10

Interpreting the Result: Here MAE and RMSE both equal 10 (this happens because all errors have the same magnitude in this specific example) — meaning predictions are, on average, off by 10 units. If errors varied more in size, RMSE would typically be somewhat larger than MAE, since it penalizes bigger mistakes more heavily.

10. Python Example

python
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_actual = np.array([100, 150, 200, 250, 300])
y_predicted = np.array([110, 140, 210, 240, 305])

mae = mean_absolute_error(y_actual, y_predicted)
mse = mean_squared_error(y_actual, y_predicted)
rmse = np.sqrt(mse)
r2 = r2_score(y_actual, y_predicted)

print("MAE:", mae)
print("MSE:", mse)
print("RMSE:", rmse)
print("R² Score:", r2)

Expected Output (approximate):

text
MAE: 9.0
MSE: 90.0
RMSE: 9.49
R² Score: 0.9862...

11. Code Explanation

• mean_absolute_error() calculates the average absolute difference between actual and predicted values.
• mean_squared_error() calculates the average SQUARED difference — notice it's larger than MAE here (90 vs 9), because squaring amplifies the effect of each individual error.
• np.sqrt(mse) calculates RMSE, bringing the metric back to the original units (dollars, points, etc.), making it easier to interpret directly than raw MSE.
• r2_score() shows that this model explains about 98.6% of the variance in the actual data — indicating a very strong fit.

12. Advantages

• Provides multiple complementary perspectives on model error (absolute vs squared, raw vs normalized).
• RMSE and MAE are both directly interpretable in the original units of the target variable.
• R² gives an intuitive, easily communicated "percentage of variance explained" summary.

13. Limitations

• MSE and RMSE are more sensitive to outliers than MAE, since large errors get squared and amplified.
• R² can sometimes be misleadingly high with overfit models (addressed further with Adjusted R² and other techniques in more advanced study).
• No single metric tells the complete story — different metrics can lead to different conclusions about which model is "better."

14. Common Mistakes

• Comparing MAE and RMSE directly as if they measure exactly the same thing — they emphasize errors differently (RMSE penalizes large errors more).
• Interpreting R² as a percentage of "accuracy" the way you might for classification — it specifically measures explained variance, a different concept.
• Not considering the presence of outliers when choosing between MAE (less sensitive) and MSE/RMSE (more sensitive).

15. Best Practices

• Report multiple regression metrics together for a complete picture, rather than relying on just one.
• Consider MAE when you want a metric less sensitive to outliers; consider RMSE/MSE when large errors should be penalized more heavily.
• Always interpret R² in the context of your specific problem — a "good" R² varies significantly by domain and application.

16. Real-World Applications

• Evaluating house price prediction models (Module 4, Topic 1/2).
• Assessing forecasting models for sales, demand, or resource planning.
• Comparing different regression algorithms or feature sets during model development.

17. Interview-Oriented Points

• Be ready to explain the difference between MAE, MSE, and RMSE, and when each might be preferred.
• Understand what R² actually measures (proportion of variance explained), and its typical range.
• Be able to explain why MSE/RMSE are more sensitive to outliers than MAE.

18. Exam-Oriented Points

• MAE = average absolute error; MSE = average squared error; RMSE = √MSE.
• R² measures the proportion of variance in the target explained by the model (0 to 1, typically).
• MSE/RMSE penalize large errors more heavily than MAE, due to squaring.

19. Comparison Table — MAE vs MSE vs RMSE

20. Quick Revision

• MAE = average absolute error; simple, directly interpretable, less sensitive to outliers.
• MSE = average squared error; penalizes large errors more heavily; used as Linear Regression's own training objective.
• RMSE = √MSE; brings the metric back to original units while still penalizing large errors more than MAE.
• R² measures the proportion of variance explained by the model, typically ranging from 0 (no explanatory power) to 1 (perfect fit).