Naive Bayes

1. Introduction

This final topic of Module 4 brings us back to the probability concepts introduced in Module 1 (Probability Basics) and Module 2 (AI & ML Fundamentals). Naive Bayes is a classification algorithm built directly on Bayes' Theorem — and despite its "naive" assumption (explained below), it performs remarkably well on many real-world problems, especially text classification.

2. What is Naive Bayes?

Simple definition: Naive Bayes is a supervised classification algorithm that predicts the class of a data point by calculating the probability of each possible class, based on Bayes' Theorem, and choosing whichever class has the highest probability.

Technical explanation: Naive Bayes applies Bayes' Theorem to classification by calculating the posterior probability of each class given the input features, under the simplifying ("naive") assumption that all features are conditionally independent of each other given the class — an assumption that's rarely perfectly true in practice, but which still yields strong, fast, and surprisingly effective classifiers.

3. Why is it Important?

• It's extremely fast to train and predict, making it practical even for very large datasets.
• It's one of the most effective and widely used algorithms specifically for text classification (like spam detection).
• It directly builds on the probability and Bayes' Theorem concepts introduced earlier in this course.

4. Prerequisites

Comfort with Probability Basics (Module 1, Topic 9), especially Bayes' Theorem.

5. Core Concepts

1. Bayes' Theorem recap
2. The "naive" conditional independence assumption
3. Calculating class probabilities
4. Types of Naive Bayes (Gaussian, Multinomial, Bernoulli)

6. Detailed Explanation

a) Bayes' Theorem Recap

Recall from Module 1: P(A|B) = [P(B|A) × P(A)] / P(B). In Naive Bayes classification, we want P(class | features) — the probability of a class, given the observed features.

b) The "Naive" Assumption

Naive Bayes assumes all features are independent of each other, given the class. For example, in email spam detection, it assumes the presence of the word "free" and the word "winner" are independent, given that the email is spam — which isn't strictly true in reality (these words likely co-occur), but this simplifying assumption makes the math dramatically easier and still works well in practice.

c) Calculating Class Probabilities

For each possible class, Naive Bayes calculates the probability of observing the given features if that class were true (multiplying together the individual probability of each feature, thanks to the independence assumption), combined with how common that class is overall (the "prior" probability). The class with the highest resulting probability is the predicted class.

d) Types of Naive Bayes

• Gaussian Naive Bayes: Assumes numeric features follow a normal (bell-curve) distribution — used for continuous data.
• Multinomial Naive Bayes: Commonly used for text classification, based on word counts/frequencies.
• Bernoulli Naive Bayes: Used for binary/boolean features (e.g., whether a specific word appears or not, regardless of count).

7. How It Works

1. Calculate the "prior" probability of each class (how common each class is in the training data overall).
2. For each feature, calculate the probability of that feature's value occurring within each class.
3. For a new data point, multiply together the prior probability and all the individual feature probabilities for each class (thanks to the independence assumption).
4. Predict whichever class produces the highest resulting probability.

8. Real-World Example

In spam email detection, Naive Bayes might learn that the word "lottery" appears in 40% of spam emails but only 0.1% of legitimate emails, and that "meeting" appears in 30% of legitimate emails but only 1% of spam emails. Given a new email containing both words, Naive Bayes combines these individual probabilities (along with the overall prior rate of spam vs legitimate emails) to calculate an overall probability for each class, and predicts whichever is higher.

9. Mathematical Explanation

Naive Bayes Classification Formula:

P(class | features) ∝ P(class) × P(feature₁ | class) × P(feature₂ | class) × ... × P(featureₙ | class)

Where:

• P(class) = the prior probability of that class occurring in the data
• P(featureᵢ | class) = the probability of observing that specific feature value, given the class
• The symbol ∝ means "proportional to" (we compare these values across classes, without needing to compute the exact normalizing denominator from Bayes' Theorem)

Numerical Example:

Suppose, based on training data:

• P(Spam) = 0.4, P(Not Spam) = 0.6
• P("free" | Spam) = 0.6, P("free" | Not Spam) = 0.1
• P("meeting" | Spam) = 0.05, P("meeting" | Not Spam) = 0.3

For a new email containing "free" but NOT "meeting":

• Score(Spam) = 0.4 × 0.6 × (1 − 0.05) = 0.4 × 0.6 × 0.95 = 0.228
• Score(Not Spam) = 0.6 × 0.1 × (1 − 0.3) = 0.6 × 0.1 × 0.7 = 0.042

Interpreting the Result: Since Score(Spam) = 0.228 is much higher than Score(Not Spam) = 0.042, Naive Bayes would classify this email as Spam. (These scores aren't final probabilities themselves, since we skipped the normalizing step, but comparing them directly is sufficient to determine the predicted class.)

10. Python Example

python
from sklearn.naive_bayes import GaussianNB
import numpy as np

# Features: [hours_studied, attendance_percent]
X = np.array([
    [1, 60], [2, 65], [3, 70], [8, 95], [9, 98], [7, 90]
])
y = np.array([0, 0, 0, 1, 1, 1])   # 0 = Fail, 1 = Pass

model = GaussianNB()
model.fit(X, y)

prediction = model.predict([[6, 85]])
print("Prediction (0=Fail, 1=Pass):", prediction[0])

probabilities = model.predict_proba([[6, 85]])
print("Probabilities [P(Fail), P(Pass)]:", probabilities[0])

Expected Output (approximate):

text
Prediction (0=Fail, 1=Pass): 1
Probabilities [P(Fail), P(Pass)]: [0.08 0.92]

11. Code Explanation

• GaussianNB() creates a Naive Bayes classifier appropriate for continuous numeric features, assuming they follow a normal distribution within each class.
• model.fit(X, y) calculates the prior probability of each class and the distribution parameters (mean, variance) for each feature within each class.
• model.predict([[6, 85]]) computes the overall probability score for each class using these learned distributions, then predicts the class with the highest score.
• model.predict_proba(...) reveals the actual normalized probabilities — here, a 92% estimated probability of "Pass" for this new student.

12. Advantages

• Extremely fast to train and predict, even on large datasets.
• Performs remarkably well on high-dimensional data, especially text classification.
• Requires relatively little training data to produce reasonable estimates, compared to many other algorithms.

13. Limitations

• The core independence assumption is rarely perfectly true in real-world data, which can limit accuracy in some cases.
• Can produce poor probability estimates (though classifications are often still correct) when the independence assumption is strongly violated.
• Gaussian Naive Bayes specifically assumes numeric features are normally distributed, which may not always hold.

14. Common Mistakes

• Assuming Naive Bayes considers feature interactions — it explicitly does NOT, due to its core independence assumption.
• Using Gaussian Naive Bayes on clearly non-normally-distributed numeric data without considering alternatives.
• Applying the wrong Naive Bayes variant (e.g., using Gaussian for word-count text data instead of Multinomial).

15. Best Practices

• Choose the appropriate Naive Bayes variant based on your data type: Gaussian (continuous), Multinomial (text/count data), or Bernoulli (binary features).
• Use Naive Bayes as a fast, strong baseline, especially for text classification problems.
• Be cautious about over-trusting exact probability estimates if you suspect the independence assumption is significantly violated — focus on the resulting classifications instead.

16. Real-World Applications

• Email spam filtering.
• Sentiment analysis (classifying text as positive/negative).
• Document/topic classification.
• Real-time prediction systems where speed is critical.

17. Interview-Oriented Points

• Be ready to explain Bayes' Theorem and how Naive Bayes applies it to classification.
• Understand the "naive" independence assumption and why it's a simplification, not a perfectly accurate reflection of reality.
• Be able to name the three common Naive Bayes variants and when each is used.

18. Exam-Oriented Points

• Naive Bayes applies Bayes' Theorem, assuming features are conditionally independent given the class.
• The predicted class is whichever has the highest combined probability score (prior × feature likelihoods).
• Variants: Gaussian (continuous data), Multinomial (text/counts), Bernoulli (binary features).

19. Comparison Table — Naive Bayes vs Logistic Regression

20. Quick Revision

• Naive Bayes classifies data by applying Bayes' Theorem, assuming features are conditionally independent given the class.
• The class with the highest combined probability score (prior × feature likelihoods) is predicted.
• Common variants: Gaussian (continuous data), Multinomial (text/word counts), Bernoulli (binary features).
• Despite its simplifying assumption, Naive Bayes is fast and performs remarkably well, especially for text classification.