Logistic Regression

1. Introduction

Despite its name, Logistic Regression is actually a classification algorithm, not a regression one — a naming quirk that trips up many beginners. It's the natural next step after Linear Regression, adapted specifically to predict categories (like "yes/no" or "spam/not spam") instead of continuous numbers.

2. What is Logistic Regression?

Simple definition: Logistic Regression is a supervised learning algorithm used for classification — it predicts the probability that an input belongs to a particular category, most commonly for two-class (binary) problems.

Technical explanation: Logistic Regression models the probability of a binary outcome by applying the sigmoid function to a linear combination of input features, squashing the output into a range between 0 and 1, which is then converted into a class prediction using a decision threshold (commonly 0.5).

3. Why is it Important?

• It's the most widely used baseline algorithm for binary classification problems in the real world.
• It's simple, fast, and highly interpretable — coefficients can be understood in terms of how they affect the probability of the outcome.
• It introduces the sigmoid function and probability-based thinking, concepts that carry forward into Neural Networks (Module 10).

4. Prerequisites

Comfort with Linear Regression (Topic 1) and Probability Basics (Module 1, Topic 9).

5. Core Concepts

1. The sigmoid function
2. From linear combination to probability
3. The decision threshold
4. Binary classification with Logistic Regression

6. Detailed Explanation

a) The Sigmoid Function

The sigmoid function takes any real number and "squashes" it into a range between 0 and 1, making it perfect for representing a probability. Its S-shaped curve means very negative inputs get pushed close to 0, very positive inputs get pushed close to 1, and inputs near 0 fall near the middle (0.5).

b) From Linear Combination to Probability

Logistic Regression first computes a linear combination of the input features (just like Linear Regression: z = b₀ + b₁x₁ + ... + bₙxₙ), then passes that result z through the sigmoid function to get a probability between 0 and 1.

c) The Decision Threshold

Once we have a probability, we need to convert it into an actual class prediction. By default, a threshold of 0.5 is used: probability ≥ 0.5 → predict class 1; probability < 0.5 → predict class 0. This threshold can be adjusted depending on the specific problem's needs (explored further in Module 6).

d) Binary Classification

Logistic Regression is most naturally suited to two-class (binary) problems, like spam/not spam or pass/fail, though extensions exist for multi-class problems too.

7. How It Works

1. Collect labeled training data with a binary target (0 or 1).
2. The algorithm learns coefficients (b₀, b₁, ..., bₙ) that best separate the two classes, by minimizing a cost function suited to probabilities (called Log Loss, rather than MSE).
3. For a new input, compute z = b₀ + b₁x₁ + ... + bₙxₙ, then apply the sigmoid function to get a probability.
4. Apply the decision threshold (commonly 0.5) to output a final class prediction.

8. Real-World Example

Predicting whether a customer will churn (cancel their subscription) based on their usage patterns. Logistic Regression doesn't just say "yes" or "no" — it estimates a probability (e.g., "there's a 78% chance this customer will churn"), which is often more useful for business decision-making than a blunt yes/no answer, since it lets the business prioritize which customers to focus retention efforts on.

9. Mathematical Explanation

Linear Combination:

z = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ

Sigmoid Function:

P(y=1) = 1 / (1 + e^(−z))

Where:

• P(y=1) = predicted probability that the output belongs to class 1
• e = Euler's number (≈ 2.718)
• z = the linear combination computed above

Decision Rule:

If P(y=1) ≥ 0.5 → predict class 1 If P(y=1) < 0.5 → predict class 0

Numerical Example:

Suppose a trained model learns: z = -4 + 0.05 × hours_studied

For a student who studied 100 hours: z = -4 + (0.05 × 100) = -4 + 5 = 1

P(y=1) = 1 / (1 + e^(-1)) = 1 / (1 + 0.368) = 1 / 1.368 ≈ 0.731

Interpreting the Result: Since 0.731 ≥ 0.5, the model predicts class 1 (e.g., "will pass"), with an estimated 73.1% probability of passing.

10. Python Example

python
import numpy as np
from sklearn.linear_model import LogisticRegression

# Features: hours studied; Labels: 0 = Fail, 1 = Pass
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

model = LogisticRegression()
model.fit(X, y)

# Predicting class labels
predictions = model.predict([[3.5], [6.5]])
print("Predictions (0=Fail, 1=Pass):", predictions)

# Predicting probabilities
probabilities = model.predict_proba([[3.5], [6.5]])
print("Probabilities [P(Fail), P(Pass)]:")
print(probabilities)

Expected Output (approximate):

text
Predictions (0=Fail, 1=Pass): [0 1]
Probabilities [P(Fail), P(Pass)]:
[[0.62 0.38]
 [0.09 0.91]]

11. Code Explanation

• LogisticRegression() creates the model; .fit(X, y) trains it by finding the coefficients that best separate the two classes.
• model.predict([[3.5], [6.5]]) returns hard class predictions (0 or 1) based on the default 0.5 threshold.
• model.predict_proba(...) returns the actual underlying probabilities for each class — notice the student who studied 3.5 hours has a 38% chance of passing (below the 0.5 threshold, hence predicted "Fail"), while the student who studied 6.5 hours has a 91% chance (comfortably above 0.5, predicted "Pass").
• This distinction between .predict() and .predict_proba() is important: the probabilities often carry more nuanced information than the final hard classification alone.

12. Advantages

• Simple, fast, and highly interpretable, similar to Linear Regression.
• Outputs meaningful probabilities, not just hard class labels, useful for risk-based decision-making.
• Works well as a strong baseline for many binary classification problems.

13. Limitations

• Assumes a roughly linear relationship between the features and the log-odds of the outcome, which can limit performance on complex, non-linear problems.
• Primarily designed for binary classification (multi-class extensions exist but add complexity).
• Sensitive to highly correlated features (multicollinearity), similar to Linear Regression.

14. Common Mistakes

• Confusing Logistic Regression with Linear Regression due to the similar name — remember, it's used for classification, not predicting continuous values.
• Forgetting that .predict() uses a default 0.5 threshold, which might not be appropriate for every problem (e.g., medical diagnoses might need a lower threshold to avoid missing true positive cases).
• Not scaling features before training, which can affect convergence and interpretation of coefficients.

15. Best Practices

• Use .predict_proba() when the actual probability estimate matters more than just a hard classification.
• Scale numeric features (Module 3, Topic 7) before training, especially when using regularization (Module 7).
• Adjust the decision threshold thoughtfully based on the specific costs of false positives vs false negatives in your problem.

16. Real-World Applications

• Email spam detection.
• Medical diagnosis (e.g., predicting presence/absence of a disease).
• Customer churn prediction.
• Credit approval (predicting loan default risk).

17. Interview-Oriented Points

• Be ready to explain why Logistic Regression is a classification algorithm despite its name.
• Understand the role of the sigmoid function in converting a linear combination into a probability.
• Be able to explain the difference between .predict() and .predict_proba().

18. Exam-Oriented Points

• Logistic Regression predicts probabilities using the sigmoid function: P(y=1) = 1 / (1 + e^(-z)).
• The default decision threshold is 0.5, converting probabilities into class predictions.
• It's primarily used for binary classification problems.

19. Comparison Table — Linear Regression vs Logistic Regression

20. Quick Revision

• Logistic Regression is a classification algorithm (despite the name), used mainly for binary classification.
• It applies the sigmoid function to a linear combination of features, producing a probability between 0 and 1.
• A decision threshold (commonly 0.5) converts this probability into a final class prediction.
• .predict() gives class labels; .predict_proba() gives underlying probabilities.