ROC-AUC Curve

1. Introduction

So far, we've evaluated classification models at a single, fixed decision threshold (usually 0.5, as introduced with Logistic Regression in Module 4). The ROC-AUC curve takes a broader view — showing how a model performs across EVERY possible threshold, giving a fuller picture of its discriminative ability.

2. What is the ROC-AUC Curve?

Simple definition: The ROC (Receiver Operating Characteristic) curve is a graph showing how a classification model's True Positive Rate and False Positive Rate change as the decision threshold varies; AUC (Area Under the Curve) summarizes this entire curve into a single number representing overall model quality.

Technical explanation: The ROC curve plots the True Positive Rate (Recall) against the False Positive Rate at every possible classification threshold, and the AUC calculates the area under this curve — a value between 0 and 1, where 1.0 represents a perfect classifier and 0.5 represents performance no better than random guessing.

3. Why is it Important?

• It evaluates a model's performance across ALL thresholds, not just the default 0.5, giving a threshold-independent view of model quality.
• AUC provides a single, easily comparable number for ranking multiple models against each other.
• It's especially useful for imbalanced classification problems, complementing Precision/Recall.

4. Prerequisites

Comfort with Logistic Regression (Module 4, Topic 3), Accuracy/Precision/Recall (Topic 1), and Confusion Matrix (Topic 2).

5. Core Concepts

1. True Positive Rate (TPR) and False Positive Rate (FPR)
2. How the ROC curve is built by varying the threshold
3. AUC (Area Under the Curve)
4. Interpreting AUC values

6. Detailed Explanation

a) True Positive Rate (TPR) and False Positive Rate (FPR)

TPR (also called Recall or Sensitivity) measures how many actual positives were correctly identified. FPR measures how many actual negatives were incorrectly flagged as positive.

b) Building the ROC Curve

Rather than using a single fixed threshold (like 0.5), the ROC curve is built by calculating TPR and FPR at MANY different threshold values (from very low to very high), plotting each resulting (FPR, TPR) point, and connecting them into a curve.

c) AUC (Area Under the Curve)

AUC condenses the entire ROC curve into a single number — the area underneath it. A perfect classifier has an AUC of 1.0 (the curve hugs the top-left corner); a random-guessing classifier has an AUC of 0.5 (a diagonal line from bottom-left to top-right).

d) Interpreting AUC Values

• AUC = 1.0: Perfect classifier.
• AUC = 0.5: No better than random guessing.
• AUC < 0.5: Worse than random (rare, and often indicates a labeling or implementation issue).
• Generally, higher AUC indicates the model does a better job of ranking positive cases above negative cases, across all possible thresholds.

7. How It Works

1. Train a classification model that outputs probabilities (not just hard class labels).
2. For many different threshold values (e.g., 0.1, 0.2, ..., 0.9), calculate TPR and FPR using that threshold.
3. Plot each (FPR, TPR) pair, connecting them to form the ROC curve.
4. Calculate the area under this curve (AUC) as a single summary metric.

8. Real-World Example

Imagine comparing two different spam filters. Filter A has an AUC of 0.95, Filter B has an AUC of 0.75. This tells you that, across ALL possible sensitivity settings (not just one specific threshold), Filter A is consistently much better at distinguishing spam from legitimate email than Filter B — a more complete comparison than looking at accuracy at just one threshold.

9. Mathematical Explanation

True Positive Rate (TPR) Formula:

TPR = TP / (TP + FN)

(Note: this is identical to Recall from Topic 1.)

False Positive Rate (FPR) Formula:

FPR = FP / (FP + TN)

Numerical Example:

At a specific threshold, suppose: TP=40, FN=10, FP=15, TN=35

• TPR = 40/(40+10) = 40/50 = 0.80
• FPR = 15/(15+35) = 15/50 = 0.30

This gives one point on the ROC curve: (FPR=0.30, TPR=0.80). Repeating this calculation at many different thresholds produces the full curve, and the AUC is the total area beneath all these connected points.

Interpreting the Result: This particular point shows that at this threshold, the model correctly catches 80% of actual positives while incorrectly flagging 30% of actual negatives — the ROC curve as a whole shows this tradeoff across every possible threshold choice.

10. Python Example

python
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

model = LogisticRegression()
model.fit(X, y)

# Getting predicted probabilities (needed for ROC curve, not just hard predictions)
y_probabilities = model.predict_proba(X)[:, 1]

fpr, tpr, thresholds = roc_curve(y, y_probabilities)
auc_score = roc_auc_score(y, y_probabilities)

print("AUC Score:", auc_score)

plt.plot(fpr, tpr, label=f"ROC Curve (AUC = {auc_score:.2f})")
plt.plot([0, 1], [0, 1], linestyle="--", color="gray", label="Random Guessing")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.show()

Expected Output (approximate):

text
AUC Score: 1.0

(A chart also displays, showing the ROC curve alongside a diagonal reference line for random guessing.)

11. Code Explanation

• model.predict_proba(X)[:, 1] retrieves the predicted PROBABILITY of the positive class for each example — essential for building the ROC curve, since we need to test many different thresholds, not just one.
• roc_curve(y, y_probabilities) calculates FPR and TPR at many different threshold values automatically.
• roc_auc_score(y, y_probabilities) calculates the single AUC summary value.
• The plotted diagonal line represents what a random-guessing classifier's ROC curve would look like (AUC=0.5) — the further the model's actual curve bulges toward the top-left corner (away from this diagonal), the better it's performing.

12. Advantages

• Evaluates model performance across all possible thresholds, not just one fixed choice.
• AUC provides an easy, single-number way to compare multiple models.
• Particularly useful and informative for imbalanced classification problems.

13. Limitations

• Requires predicted probabilities, not just hard class labels — some algorithms don't naturally provide these.
• AUC can sometimes be overly optimistic on VERY imbalanced datasets, where Precision-Recall curves may give a more informative picture.
• A single AUC number doesn't tell you which SPECIFIC threshold to actually use in practice — that decision still requires additional judgment.

14. Common Mistakes

• Using .predict() (hard labels) instead of .predict_proba() (probabilities) when calculating ROC-AUC — this produces an incorrect, overly simplistic curve.
• Assuming a high AUC always means a model is ready for deployment, without also checking Precision/Recall at the actual threshold you plan to use.
• Confusing AUC with Accuracy — they measure fundamentally different things.

15. Best Practices

• Always use predicted probabilities (predict_proba()), not hard class predictions, when building ROC curves.
• Use AUC to compare different models' overall discriminative ability, but still choose a specific threshold thoughtfully based on your problem's Precision/Recall needs.
• For very imbalanced datasets, consider examining a Precision-Recall curve alongside the ROC curve for a fuller picture.

16. Real-World Applications

• Comparing multiple candidate models before choosing one for deployment.
• Medical test evaluation (e.g., comparing diagnostic tests' overall discriminative power).
• Credit scoring model comparison in the finance industry.

17. Interview-Oriented Points

• Be ready to explain what TPR and FPR represent, and how the ROC curve is constructed by varying the threshold.
• Understand what AUC values of 1.0, 0.5, and below 0.5 each indicate.
• Be able to explain why ROC-AUC evaluates a model across all thresholds, unlike a single Accuracy or F1 score.

18. Exam-Oriented Points

• ROC curve plots TPR (y-axis) vs FPR (x-axis) across all possible thresholds.
• AUC = Area Under the (ROC) Curve; ranges from 0 to 1, with 0.5 representing random guessing.
• Requires predicted probabilities, not just hard class predictions.

19. Comparison Table — ROC-AUC vs Precision-Recall Curve

20. Quick Revision

• The ROC curve plots True Positive Rate against False Positive Rate across all possible classification thresholds.
• AUC condenses this curve into a single number: 1.0 = perfect, 0.5 = random guessing.
• Requires predicted probabilities (predict_proba()), not just hard predictions.
• For very imbalanced datasets, consider a Precision-Recall curve alongside ROC-AUC for a fuller picture.