Regularization (L1/L2)

1. Introduction

Now that you understand overfitting (Topic 1), let's explore one of the most powerful and widely used techniques for directly combating it: Regularization. Applied primarily to linear models (Linear Regression, Logistic Regression from Module 4), regularization works by discouraging a model from assigning overly large weights to any single feature.

2. What is Regularization?

Simple definition: Regularization is a technique that reduces overfitting by adding a penalty for large coefficient (weight) values, encouraging the model to stay simpler and more general.

Technical explanation: Regularization modifies a model's cost function by adding a penalty term based on the magnitude of its coefficients — L1 regularization (Lasso) adds a penalty proportional to the ABSOLUTE VALUE of coefficients, which can shrink some coefficients to exactly zero (performing automatic feature selection), while L2 regularization (Ridge) adds a penalty proportional to the SQUARED value of coefficients, shrinking all coefficients toward zero without necessarily eliminating any entirely.

3. Why is it Important?

• It's one of the most direct, effective, and widely used tools for combating overfitting (Topic 1) in linear models.
• L1 regularization can automatically perform feature selection, simplifying models and improving interpretability.
• It's a standard technique used across countless real-world ML pipelines, especially with many features.

4. Prerequisites

Comfort with Linear Regression/Logistic Regression (Module 4) and Overfitting (Topic 1).

5. Core Concepts

1. The regularization penalty term
2. L1 Regularization (Lasso)
3. L2 Regularization (Ridge)
4. The regularization strength hyperparameter (alpha/lambda)

6. Detailed Explanation

a) The Regularization Penalty Term

Normally, Linear Regression minimizes just the Mean Squared Error (Module 4, Topic 1). Regularization adds an EXTRA penalty term to this cost function, based on the size of the model's coefficients — the larger the coefficients, the bigger this penalty, discouraging the model from relying too heavily on any single feature.

b) L1 Regularization (Lasso)

L1 (Lasso — Least Absolute Shrinkage and Selection Operator) adds a penalty proportional to the SUM OF ABSOLUTE VALUES of the coefficients. A key property of L1 is that it can shrink some coefficients all the way to EXACTLY ZERO, effectively removing those features from the model entirely — a form of automatic feature selection.

c) L2 Regularization (Ridge)

L2 (Ridge) adds a penalty proportional to the SUM OF SQUARED VALUES of the coefficients. Unlike L1, it shrinks coefficients toward zero but rarely eliminates them completely — all features typically remain in the model, just with smaller, more conservative weights.

d) The Regularization Strength (alpha/lambda)

A hyperparameter (commonly called alpha in Scikit-learn, or lambda in some textbooks) controls how STRONG the regularization penalty is. A larger alpha means stronger regularization (simpler model, more shrinkage); alpha=0 means no regularization at all (equivalent to standard Linear Regression).

7. How It Works

1. Start with the normal cost function (e.g., MSE for Linear Regression).
2. Add a penalty term based on the coefficients' magnitudes — sum of absolute values (L1) or sum of squared values (L2).
3. Multiply this penalty by the chosen alpha (regularization strength).
4. The algorithm now minimizes: original error + regularization penalty, naturally favoring solutions with smaller, more conservative coefficients.

8. Real-World Example

Imagine predicting house prices using 50 different features, many of which are only weakly related to price (like "number of light switches"). Without regularization, Linear Regression might assign small but non-zero weights to all 50 features, some based purely on training-data coincidence. L1 regularization would likely shrink the truly irrelevant features' weights all the way to zero, automatically simplifying the model to focus only on genuinely important features like size, location, and age.

9. Mathematical Explanation

L1 Regularization (Lasso) Cost Function:

Cost = MSE + α × Σ|wᵢ|

L2 Regularization (Ridge) Cost Function:

Cost = MSE + α × Σ(wᵢ)²

Where:

• MSE = the original Mean Squared Error (Module 4, Topic 1)
• wᵢ = each individual model coefficient (weight)
• α (alpha) = the regularization strength hyperparameter
• Σ = sum across all coefficients

Numerical Example:

Suppose a model has learned coefficients: w₁=4, w₂=0.1, w₃=3, with α=0.1

• L1 penalty = 0.1 × (|4| + |0.1| + |3|) = 0.1 × 7.1 = 0.71
• L2 penalty = 0.1 × (4² + 0.1² + 3²) = 0.1 × (16+0.01+9) = 0.1 × 25.01 = 2.501

Interpreting the Result: These penalty values are ADDED to the original MSE during training, encouraging the algorithm to find a balance between fitting the data well AND keeping coefficients small — notice how L2's penalty grows much faster for large coefficients (like w₁=4) due to squaring, while L1 treats all coefficient magnitudes more proportionally.

10. Python Example

python
from sklearn.linear_model import Ridge, Lasso, LinearRegression
import numpy as np

# Features: 4 columns, some likely more relevant than others
X = np.array([
    [1, 50, 2, 100], [2, 55, 3, 105], [3, 60, 1, 98],
    [4, 65, 4, 110], [5, 70, 2, 102], [6, 75, 5, 115]
])
y = np.array([200, 220, 210, 240, 225, 250])

linear_model = LinearRegression().fit(X, y)
ridge_model = Ridge(alpha=10).fit(X, y)
lasso_model = Lasso(alpha=1.0).fit(X, y)

print("Linear Regression coefficients:", linear_model.coef_)
print("Ridge (L2) coefficients:       ", ridge_model.coef_)
print("Lasso (L1) coefficients:       ", lasso_model.coef_)

Expected Output (approximate — exact values depend on the data):

text
Linear Regression coefficients: [ 5.2  0.8 -1.1  0.3]
Ridge (L2) coefficients:        [ 3.1  0.6 -0.4  0.2]
Lasso (L1) coefficients:        [ 2.8  0.5  0.   0. ]

11. Code Explanation

• LinearRegression() (no regularization) shows the "raw" learned coefficients, which may be large or unstable, especially with correlated features.
• Ridge(alpha=10) shrinks all coefficients toward zero, but keeps every feature in the model with a non-zero (though smaller) weight.
• Lasso(alpha=1.0) goes further — notice how it shrinks two coefficients ALL THE WAY to exactly 0, effectively removing those features from the model entirely, demonstrating L1's automatic feature-selection property.
• Comparing all three side by side shows exactly how regularization changes (and typically shrinks/simplifies) the learned model.

12. Advantages

• Directly and effectively reduces overfitting in linear models.
• L1 (Lasso) performs automatic feature selection, improving interpretability and simplicity.
• L2 (Ridge) handles multicollinearity (Module 4, Topic 2) better than unregularized Linear Regression.

13. Limitations

• Requires tuning the alpha hyperparameter (too high = underfitting; too low = insufficient regularization) — typically done via Hyperparameter Tuning (Topic 3).
• L1's feature-elimination behavior can be somewhat unpredictable/unstable with highly correlated features.
• Regularization assumes features are on a comparable scale — feature scaling (Module 3, Topic 7) is essential beforehand.

14. Common Mistakes

• Applying regularization without first scaling features, which can cause the penalty to unfairly affect features with different natural scales.
• Choosing alpha arbitrarily without using Cross-Validation or Hyperparameter Tuning (Topic 3) to find a good value.
• Confusing L1 and L2 — remember, L1 (Lasso) can zero out coefficients; L2 (Ridge) generally does not.

15. Best Practices

• Always scale features (Module 3, Topic 7) before applying regularization.
• Use Cross-Validation (Module 6, Topic 4) combined with Hyperparameter Tuning (Topic 3) to find a good alpha value.
• Consider L1 (Lasso) when you suspect many features are irrelevant and want automatic feature selection; consider L2 (Ridge) when you want to retain all features but reduce their influence more conservatively.

16. Real-World Applications

• Predicting outcomes from datasets with many potentially irrelevant features (e.g., genomics, text data).
• Improving the stability and generalization of Linear/Logistic Regression models in production systems.
• Automatically simplifying models for easier interpretation and deployment (via L1's feature elimination).

17. Interview-Oriented Points

• Be ready to explain the difference between L1 and L2 regularization, including their formulas and key behavioral difference (zeroing out coefficients vs. not).
• Understand the role of the alpha hyperparameter in controlling regularization strength.
• Be able to explain why regularization helps combat overfitting.

18. Exam-Oriented Points

• L1 (Lasso): penalty = α × Σ|wᵢ|; can shrink coefficients to exactly zero (automatic feature selection).
• L2 (Ridge): penalty = α × Σ(wᵢ)²; shrinks coefficients toward zero but rarely to exactly zero.
• alpha controls regularization strength; higher alpha = simpler model, more shrinkage.

19. Comparison Table — L1 (Lasso) vs L2 (Ridge) Regularization

20. Quick Revision

• Regularization adds a penalty for large coefficients to a model's cost function, reducing overfitting.
• L1 (Lasso): penalty based on absolute coefficient values; can zero out coefficients (automatic feature selection).
• L2 (Ridge): penalty based on squared coefficient values; shrinks coefficients but rarely to exactly zero.
• alpha controls regularization strength — tune it via Cross-Validation and Hyperparameter Tuning.