Hyperparameter Tuning

1. Introduction

Throughout this course, you've encountered many hyperparameters — k in KNN, max_depth in Decision Trees, alpha in Ridge/Lasso, n_estimators in Random Forest. But how do you actually CHOOSE good values for these, rather than guessing? This topic covers the systematic techniques for finding strong hyperparameter values: Grid Search and Random Search.

2. What is Hyperparameter Tuning?

Simple definition: Hyperparameter Tuning is the process of systematically searching for the combination of hyperparameter values that produces the best-performing model.

Technical explanation: Hyperparameter Tuning involves defining a search space of candidate hyperparameter values, evaluating model performance (typically via Cross-Validation, Module 6, Topic 4) for each candidate combination, and selecting the combination that yields the best validated performance — commonly performed through Grid Search (exhaustively trying every combination) or Random Search (sampling a random subset of combinations).

3. Why is it Important?

• Default hyperparameter values are rarely optimal for every specific dataset — tuning can meaningfully improve model performance.
• It directly connects to Overfitting/Underfitting (Topic 1) — many hyperparameters (like max_depth, alpha, k) directly control model complexity.
• It's a standard, expected step in nearly every serious real-world ML project.

4. Prerequisites

Comfort with Model, Parameters & Hyperparameters (Module 2, Topic 10) and Cross-Validation (Module 6, Topic 4).

5. Core Concepts

1. The hyperparameter search space
2. Grid Search (exhaustive search)
3. Random Search (randomized sampling)
4. Combining tuning with Cross-Validation

6. Detailed Explanation

a) The Hyperparameter Search Space

Before tuning, you define a "search space" — the range or set of candidate values you want to try for each hyperparameter (e.g., max_depth = [3, 5, 7, 10], n_estimators = [50, 100, 200]).

b) Grid Search

Grid Search exhaustively tries EVERY possible combination of the specified hyperparameter values, evaluating each one (typically via Cross-Validation) and reporting the best-performing combination. It's thorough but can become very computationally expensive as the number of hyperparameters and candidate values grows.

c) Random Search

Random Search instead samples a RANDOM subset of combinations from the search space (you specify how many combinations to try), rather than testing every single one. Surprisingly, this often finds a comparably good combination much faster than Grid Search, especially when only a few hyperparameters actually matter significantly.

d) Combining with Cross-Validation

Both Grid Search and Random Search typically evaluate each candidate hyperparameter combination using Cross-Validation (not just a single train-test split), ensuring the chosen "best" combination is robust and not just a lucky fit to one particular split.

7. How It Works

1. Define the hyperparameter search space (which values to try for each hyperparameter).
2. Choose Grid Search (exhaustive) or Random Search (sampled) based on your computational budget and search space size.
3. For each candidate combination, evaluate performance using Cross-Validation.
4. Select the combination with the best average Cross-Validation performance.
5. Train a final model using this best combination on the full training data.

8. Real-World Example

Imagine tuning a Random Forest's n_estimators (number of trees) and max_depth (tree depth) together. Grid Search might try every combination from n_estimators=[50,100,200] and max_depth=[5,10,15] — that's 9 total combinations, each evaluated via Cross-Validation, before selecting whichever combination performed best on average. Random Search, by contrast, might randomly sample just 5 of these 9 combinations (or draw from continuous ranges), often finding a similarly good result much faster.

9. Mathematical Explanation

Hyperparameter Tuning doesn't introduce new formulas itself — it applies Cross-Validation's mean scoring (Module 6, Topic 4) repeatedly across many hyperparameter combinations.

Conceptual Search Space Size (Grid Search):

Total Combinations = (number of values for hyperparameter 1) × (number of values for hyperparameter 2) × ...

Numerical Example:

Suppose you're tuning 2 hyperparameters: max_depth with 4 candidate values, and min_samples_split with 3 candidate values.

Total Grid Search combinations = 4 × 3 = 12

If using 5-Fold Cross-Validation for each combination, the TOTAL number of model training runs = 12 × 5 = 60

Interpreting the Result: This illustrates why Grid Search can become computationally expensive quickly — adding just one more hyperparameter with 3 more candidate values would TRIPLE the total combinations (and training runs) required, which is exactly why Random Search becomes attractive for larger search spaces.

10. Python Example

python
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
import numpy as np

X = np.array([[1,60],[2,65],[3,70],[8,95],[9,98],[7,90],[4,72],[6,88],[5,80],[10,99]])
y = np.array([0,0,0,1,1,1,0,1,0,1])

# Grid Search
param_grid = {"n_estimators": [50, 100], "max_depth": [2, 4, None]}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42), param_grid, cv=3)
grid_search.fit(X, y)

print("Best parameters (Grid Search):", grid_search.best_params_)
print("Best CV score (Grid Search):", grid_search.best_score_)

# Random Search
random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42), param_grid, n_iter=3, cv=3, random_state=42
)
random_search.fit(X, y)

print("\nBest parameters (Random Search):", random_search.best_params_)
print("Best CV score (Random Search):", random_search.best_score_)

Expected Output (approximate):

text
Best parameters (Grid Search): {'max_depth': 4, 'n_estimators': 100}
Best CV score (Grid Search): 0.83

Best parameters (Random Search): {'n_estimators': 50, 'max_depth': 4}
Best CV score (Random Search): 0.80

11. Code Explanation

• param_grid defines the search space — the candidate values to try for n_estimators and max_depth.
• GridSearchCV(...) exhaustively tries every combination (here, 2×3=6 combinations), each evaluated via 3-Fold Cross-Validation.
• grid_search.best_params_ and grid_search.best_score_ reveal the winning combination and its average Cross-Validation score.
• RandomizedSearchCV(..., n_iter=3, ...) instead randomly samples just 3 combinations out of the full search space, offering a faster (though potentially slightly less thorough) alternative.
• Both approaches automatically handle the full evaluate-compare-select process internally.

12. Advantages

• Systematically finds better-performing hyperparameter combinations than manual guessing.
• Combined with Cross-Validation, provides a robust, reliable way to select final hyperparameters.
• Random Search offers a practical, faster alternative when the search space is large.

13. Limitations

• Grid Search can become extremely computationally expensive as the number of hyperparameters and candidate values grows.
• Random Search isn't guaranteed to find the absolute best combination, since it only samples a subset.
• Both approaches still require you to manually decide the RANGE of candidate values to search over.

14. Common Mistakes

• Defining an overly large Grid Search space without considering computational cost.
• Not using Cross-Validation within the tuning process, risking overfitting to a single validation split.
• Assuming the "best" hyperparameters found on one dataset will automatically transfer perfectly to a different, unrelated dataset.

15. Best Practices

• Start with a reasonably broad but manageable search space, and narrow it down based on initial results.
• Use Random Search for large search spaces, and Grid Search for smaller, more focused ones.
• Always combine tuning with Cross-Validation for robust, reliable hyperparameter selection.
• Remember to do final evaluation on a completely separate test set, never used during the tuning process itself.

16. Real-World Applications

• Tuning Random Forest, SVM, and Gradient Boosting hyperparameters for competition-winning ML models.
• Optimizing Ridge/Lasso's alpha (Topic 2) for the best generalization performance.
• A standard step in virtually every production ML pipeline before final model deployment.

17. Interview-Oriented Points

• Be ready to explain the difference between Grid Search and Random Search.
• Understand why Cross-Validation is typically used alongside hyperparameter tuning.
• Be able to explain the computational tradeoff between thoroughness (Grid Search) and efficiency (Random Search).

18. Exam-Oriented Points

• Grid Search exhaustively tries every combination in the defined search space.
• Random Search samples a random subset of combinations, often faster for large search spaces.
• Both are typically combined with Cross-Validation for robust evaluation of each candidate combination.

19. Comparison Table — Grid Search vs Random Search

20. Quick Revision

• Hyperparameter Tuning systematically searches for the best-performing hyperparameter combination.
• Grid Search exhaustively tries every combination; Random Search samples a random subset.
• Both are typically combined with Cross-Validation for robust, reliable evaluation.
• Random Search is often preferred for large search spaces due to lower computational cost.