Accuracy, Precision, Recall, F1-Score

1. Introduction

Once you've trained a classification model (Module 4), the natural next question is: "how good is it, really?" Accuracy is the most intuitive metric, but as you'll see in this topic, it can be dangerously misleading on its own — especially with imbalanced data. Precision, Recall, and F1-Score provide a more complete, nuanced picture.

2. What are Accuracy, Precision, Recall, and F1-Score?

Simple definition: These are four different ways of measuring how well a classification model performs, each capturing a different aspect of correctness — overall accuracy, correctness of positive predictions (precision), ability to catch actual positives (recall), and a balance between the two (F1-Score).

Technical explanation: These metrics are calculated from the four possible prediction outcomes — True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) — each combining these counts in a different way to highlight a different dimension of classification performance.

3. Why is it Important?

• Accuracy alone can be dangerously misleading on imbalanced datasets (e.g., rare disease detection, fraud detection).
• Precision and Recall let you understand the specific TYPE of errors a model makes, which matters enormously depending on the real-world cost of each error type.
• These are the most commonly discussed evaluation metrics in ML interviews and real-world business reporting.

4. Prerequisites

Comfort with Supervised Learning (Module 2, Topic 5) and any classification algorithm from Module 4.

5. Core Concepts

1. True Positives, True Negatives, False Positives, False Negatives
2. Accuracy
3. Precision
4. Recall (Sensitivity)
5. F1-Score
6. Why accuracy can be misleading

6. Detailed Explanation

a) TP, TN, FP, FN

• True Positive (TP): Model correctly predicted the positive class.
• True Negative (TN): Model correctly predicted the negative class.
• False Positive (FP): Model incorrectly predicted positive (a "false alarm," also called a Type I error).
• False Negative (FN): Model incorrectly predicted negative (a "missed case," also called a Type II error).

b) Accuracy

Accuracy is simply the proportion of ALL predictions that were correct (both positive and negative). It's intuitive, but becomes misleading when classes are imbalanced.

c) Precision

Precision answers: "Of all the times the model predicted positive, how often was it actually correct?" High precision means few false alarms.

d) Recall (Sensitivity)

Recall answers: "Of all the ACTUAL positive cases, how many did the model correctly catch?" High recall means few missed cases.

e) F1-Score

F1-Score is the harmonic mean of Precision and Recall, providing a single balanced metric when you care about both false alarms and missed cases roughly equally.

f) Why Accuracy Can Be Misleading

Imagine a dataset where only 1% of transactions are fraudulent. A model that predicts "not fraud" for EVERY single transaction would achieve 99% accuracy — despite being completely useless for actually catching fraud. This is exactly why Precision and Recall matter so much for imbalanced problems.

7. How It Works

1. Make predictions on a test set and compare them to the actual known labels.
2. Count TP, TN, FP, and FN based on this comparison.
3. Calculate Accuracy, Precision, Recall, and F1-Score using these four counts.
4. Interpret the results based on which type of error matters most for your specific problem.

8. Real-World Example

For a spam email filter: a False Positive means a legitimate, important email gets marked as spam (potentially very costly — the user might miss something important). A False Negative means a spam email slips into the inbox (annoying, but usually less costly). This asymmetry means you might prioritize high Precision (avoiding false alarms) over high Recall for this particular problem — a judgment call that depends entirely on the real-world context.

9. Mathematical Explanation

Accuracy Formula:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision Formula:

Precision = TP / (TP + FP)

Recall Formula:

Recall = TP / (TP + FN)

F1-Score Formula:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Numerical Example:

Suppose a model's predictions on a test set of 100 examples result in: TP=40, TN=50, FP=5, FN=5

• Accuracy = (40+50) / 100 = 90/100 = 0.90
• Precision = 40 / (40+5) = 40/45 ≈ 0.889
• Recall = 40 / (40+5) = 40/45 ≈ 0.889
• F1 = 2 × (0.889 × 0.889) / (0.889 + 0.889) = 2 × 0.790 / 1.778 ≈ 0.889

Interpreting the Result: With 90% accuracy and roughly 89% precision/recall/F1, this model performs quite well and fairly consistently across all four metrics — a sign of balanced performance without significant class imbalance issues in this particular example.

10. Python Example

python
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Actual labels vs model predictions
y_actual =    [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_predicted = [1, 0, 1, 0, 0, 1, 0, 1, 1, 0]

print("Accuracy:", accuracy_score(y_actual, y_predicted))
print("Precision:", precision_score(y_actual, y_predicted))
print("Recall:", recall_score(y_actual, y_predicted))
print("F1-Score:", f1_score(y_actual, y_predicted))

Expected Output (approximate):

text
Accuracy: 0.8
Precision: 0.75
Recall: 0.75
F1-Score: 0.75

11. Code Explanation

• accuracy_score() compares every prediction to the actual label and calculates the overall proportion of correct predictions.
• precision_score() specifically examines how many of the model's POSITIVE predictions were actually correct.
• recall_score() specifically examines how many of the ACTUAL positive cases the model successfully identified.
• f1_score() combines precision and recall into one balanced score — notice all three (precision, recall, F1) are 0.75 here, since this particular example happens to have symmetric errors.

12. Advantages

• These metrics together give a much more complete picture of classification performance than accuracy alone.
• Precision and Recall can be prioritized differently depending on the real-world cost of different error types.
• F1-Score offers a convenient single number balancing both concerns when needed.

13. Limitations

• No single metric tells the whole story — always consider multiple metrics together, in the context of your specific problem.
• F1-Score treats Precision and Recall as equally important, which isn't always appropriate (weighted variants exist for this).
• These metrics apply most naturally to binary classification; multi-class versions require additional considerations (like averaging strategies).

14. Common Mistakes

• Relying solely on accuracy, especially with imbalanced datasets.
• Confusing Precision and Recall — remembering "Precision = correctness of positive predictions" and "Recall = coverage of actual positives" helps avoid mixing them up.
• Not considering which type of error (false positive vs false negative) is more costly for the specific real-world problem.

15. Best Practices

• Always check class balance in your dataset before deciding which metrics matter most.
• Consider the real-world cost of False Positives vs False Negatives when choosing which metric to prioritize.
• Report multiple metrics together (not just accuracy) for a complete, honest picture of model performance.

16. Real-World Applications

• Medical diagnosis (prioritizing high Recall to avoid missing true disease cases).
• Spam detection (balancing Precision to avoid blocking legitimate emails).
• Fraud detection (balancing both concerns carefully, given the costs of both missed fraud and false alarms).

17. Interview-Oriented Points

• Be ready to explain why accuracy can be misleading with imbalanced data, using a concrete example.
• Understand the precise difference between Precision and Recall.
• Be able to explain a real-world scenario where you'd prioritize Precision over Recall, or vice versa.

18. Exam-Oriented Points

• Accuracy = (TP+TN)/(TP+TN+FP+FN); Precision = TP/(TP+FP); Recall = TP/(TP+FN); F1 = harmonic mean of Precision and Recall.
• Accuracy can be misleading on imbalanced datasets.
• Precision focuses on false alarms; Recall focuses on missed cases.

19. Comparison Table — Precision vs Recall

20. Quick Revision

• Accuracy = overall proportion of correct predictions — can be misleading on imbalanced data.
• Precision = correctness of positive predictions; Recall = coverage of actual positive cases.
• F1-Score balances Precision and Recall into a single metric.
• Always choose which metric to prioritize based on the real-world cost of false positives vs false negatives.