Probability Basics

1. Introduction

Machine learning is fundamentally about making predictions under uncertainty — and probability is the mathematical language used to describe that uncertainty. Concepts like probability, conditional probability, and Bayes' theorem show up directly in algorithms like Naive Bayes, and indirectly in almost every ML model's confidence scores and decision-making.

2. What is Probability?

Simple definition: Probability is a number between 0 and 1 that describes how likely an event is to happen. A probability of 0 means the event will never happen; a probability of 1 means it will always happen.

Technical explanation: Probability is a branch of mathematics that quantifies the likelihood of an event occurring, calculated as the ratio of favorable outcomes to total possible outcomes within a defined sample space.

3. Why is it Important?

• Many ML algorithms (like Naive Bayes) are built directly on probability theory.
• Probability helps quantify uncertainty in predictions (e.g., "there's an 80% chance this email is spam").
• Understanding conditional probability is essential for grasping how models update predictions based on new information (evidence).

4. Prerequisites

Basic arithmetic and comfort with fractions and percentages. Basic Statistics (Topic 8) is helpful context, though not strictly required.

5. Core Concepts

1. Sample space and events
2. Basic probability formula
3. Mutually exclusive events
4. Independent vs dependent events
5. Conditional probability
6. Bayes' theorem (introductory)

6. Detailed Explanation

a) Sample Space and Events

The sample space is the set of all possible outcomes of an experiment (e.g., rolling a die gives a sample space of {1,2,3,4,5,6}). An event is a specific outcome or set of outcomes you're interested in (e.g., "rolling an even number" = {2,4,6}).

b) Basic Probability Formula

Probability of an event = (Number of favorable outcomes) / (Total possible outcomes).

c) Mutually Exclusive Events

Two events are mutually exclusive if they cannot both happen at the same time (e.g., getting heads and tails on a single coin flip).

d) Independent vs Dependent Events

Independent events don't affect each other's probability (e.g., two separate coin flips). Dependent events are those where one event's outcome affects the probability of the other (e.g., drawing cards from a deck without replacement).

e) Conditional Probability

Conditional probability is the probability of an event happening, given that another event has already happened. It's written as P(A | B), meaning "the probability of A, given B."

f) Bayes' Theorem (Introductory)

Bayes' theorem provides a way to update the probability of an event based on new evidence, by relating P(A | B) to P(B | A). It forms the mathematical foundation of the Naive Bayes algorithm, which you'll encounter later in this course.

7. How It Works

To calculate a simple probability:

1. Identify the total sample space (all possible outcomes).
2. Identify the favorable outcomes (the specific event you care about).
3. Divide favorable outcomes by total outcomes.

For conditional probability:

1. Narrow the sample space down to only the outcomes where the "given" condition is true.
2. Within that narrowed space, calculate the probability of the event you're interested in.

8. Real-World Example

Imagine checking the weather forecast. Normally, the chance of rain tomorrow might be 20%. But if you're told "it's currently cloudy and humid" (new evidence), your updated (conditional) probability of rain might rise to 60%. This everyday intuition — updating your belief based on new information — is exactly what conditional probability and Bayes' theorem formalize mathematically.

9. Technical Example

Rolling a fair six-sided die:

• Sample space = {1, 2, 3, 4, 5, 6}
• Event "rolling an even number" = {2, 4, 6}
• Probability = 3/6 = 0.5

10. Mathematical Explanation

Basic Probability Formula:

P(A) = (Number of favorable outcomes for A) / (Total number of possible outcomes)

Conditional Probability Formula:

P(A | B) = P(A and B) / P(B)

Where:

• P(A | B) = probability of A occurring, given B has occurred
• P(A and B) = probability that both A and B occur
• P(B) = probability of B occurring (must be greater than 0)

Bayes' Theorem Formula:

P(A | B) = [P(B | A) × P(A)] / P(B)

Where:

• P(A | B) = probability of A given B (what we want to find)
• P(B | A) = probability of B given A
• P(A) = probability of A occurring on its own
• P(B) = probability of B occurring on its own

Numerical Example:

Suppose in a class of 100 students: 40 students study Math, 30 students study Science, and 10 students study both.

• P(Math) = 40/100 = 0.4
• P(Science) = 30/100 = 0.3
• P(Math and Science) = 10/100 = 0.1

P(Science | Math) = P(Math and Science) / P(Math) = 0.1 / 0.4 = 0.25

Interpreting the Result: Given that a student studies Math, there's a 25% chance they also study Science — even though, overall, 30% of all students study Science. This shows how conditional probability can differ from the plain (unconditional) probability.

11. Python Example

python
import random

# Simulating probability of rolling an even number on a fair die
total_rolls = 10000
even_count = 0

for _ in range(total_rolls):
    roll = random.randint(1, 6)
    if roll % 2 == 0:
        even_count += 1

probability_even = even_count / total_rolls
print(f"Estimated probability of rolling an even number: {probability_even:.2f}")

# Simulating conditional probability example (Math and Science students)
students = []
for _ in range(100):
    studies_math = random.random() < 0.4
    if studies_math:
        studies_science = random.random() < 0.25   # roughly reflecting the earlier example
    else:
        studies_science = random.random() < 0.33
    students.append((studies_math, studies_science))

math_students = [s for s in students if s[0]]
math_and_science = [s for s in math_students if s[1]]

if len(math_students) > 0:
    conditional_prob = len(math_and_science) / len(math_students)
    print(f"Estimated P(Science | Math): {conditional_prob:.2f}")

Expected Output (values will vary slightly due to randomness):

text
Estimated probability of rolling an even number: 0.50
Estimated P(Science | Math): 0.24

12. Code Explanation

• random.randint(1, 6) simulates rolling a fair six-sided die.
• The loop repeats this simulation 10,000 times, counting how often the result is even — dividing by the total gives an estimated probability, which should be close to the true theoretical probability of 0.5.
• In the second part, random.random() < 0.4 simulates a 40% chance of a student studying Math, mimicking real probability through random simulation.
• Filtering math_students and then math_and_science mirrors the conditional probability formula: we narrow down to only Math students, then check how many of them also study Science.
• This kind of simulation-based approach (called a "Monte Carlo" method) is a common way to estimate probabilities that might be hard to calculate directly.

13. Advantages

• Provides a rigorous way to quantify and reason about uncertainty.
• Forms the mathematical basis for probabilistic ML algorithms like Naive Bayes.
• Helps in understanding model confidence and decision-making under uncertainty.

14. Limitations

• Probability calculations assume you have accurate information about the sample space and event probabilities, which isn't always realistic.
• Conditional probability and Bayes' theorem can be conceptually tricky for beginners at first.
• Small sample sizes can give unreliable probability estimates.

15. Common Mistakes

• Confusing P(A | B) with P(B | A) — these are generally NOT the same value.
• Assuming independence between events when they are actually dependent (a very common real-world error).
• Forgetting that probabilities must be between 0 and 1.
• Mixing up "mutually exclusive" (cannot happen together) with "independent" (don't affect each other) — these are different concepts.

16. Best Practices

• Always clearly define the sample space before calculating any probability.
• Double-check whether events are truly independent before applying simplified formulas.
• Use simulations (like the Python example above) to sanity-check theoretical probability calculations.
• Be extra careful with conditional probability — always be clear about which event is "given."

17. Real-World Applications

• Spam email detection using the Naive Bayes algorithm, which relies directly on conditional probability.
• Medical diagnosis systems that update disease probability based on test results (a classic Bayes' theorem application).
• Recommendation systems estimating the probability a user will like a certain product.

18. Interview-Oriented Points

• Be ready to explain the difference between independent and mutually exclusive events with an example.
• Understand and be able to state Bayes' theorem, and briefly explain what each term means.
• Know why conditional probability P(A|B) is generally different from P(B|A).
• Be able to connect probability concepts to how Naive Bayes works at a high level.

19. Exam-Oriented Points

• P(A) = favorable outcomes / total outcomes.
• Mutually exclusive events cannot occur together; independent events don't affect each other's probability.
• Conditional probability: P(A | B) = P(A and B) / P(B).
• Bayes' theorem: P(A | B) = [P(B | A) × P(A)] / P(B).

20. Comparison Table — Mutually Exclusive vs Independent Events

21. Quick Revision

• Probability = favorable outcomes / total outcomes, always between 0 and 1.
• Mutually exclusive events cannot happen together; independent events don't influence each other.
• Conditional probability P(A | B) narrows the sample space to cases where B has already happened.
• Bayes' theorem lets us "flip" conditional probabilities: relating P(A|B) to P(B|A).
• These concepts form the mathematical foundation of the Naive Bayes ML algorithm.