Model, Parameters & Hyperparameters

1. Introduction

This final topic of Module 2 clarifies three words you'll see constantly throughout the rest of this course: model, parameters, and hyperparameters. They sound similar but mean very different things — and mixing them up is one of the most common sources of confusion for ML beginners.

2. What are Model, Parameters, and Hyperparameters?

Simple definition: A model is the overall system that makes predictions. Parameters are the internal values the model learns automatically from data during training. Hyperparameters are settings that a human chooses before training begins, controlling how the model learns.

Technical explanation: A model is the mathematical structure (and its learned state) produced by a training algorithm. Parameters are the internal variables (such as weights and biases) whose values are adjusted automatically during training to minimize error. Hyperparameters are configuration values set prior to training (such as the number of neighbors in KNN, or the maximum depth of a decision tree) that control the training process itself but are not learned from the data.

3. Why is it Important?

• Nearly every algorithm you'll study in Module 4 involves both parameters (learned automatically) and hyperparameters (set manually).
• Hyperparameter tuning (covered in depth in Module 7) is a critical skill for improving model performance.
• This distinction is one of the most common interview questions for ML roles.

4. Prerequisites

Comfort with Topic 2 (What is Machine Learning?) and Topic 5 (Supervised Learning).

5. Core Concepts

1. Model (the trained system)
2. Parameters (learned automatically during training)
3. Hyperparameters (set manually before training)
4. How parameters and hyperparameters interact during training

6. Detailed Explanation

a) Model

A model is the complete, trained result of an ML algorithm — it "contains" everything the algorithm learned and is what you actually use to make predictions on new data.

b) Parameters

Parameters are values inside the model that are automatically adjusted during training. For example, in Linear Regression, the slope and intercept of the line are parameters — the algorithm discovers their best values by analyzing the training data.

c) Hyperparameters

Hyperparameters are settings you (the developer) choose before training starts, and they are not learned from data. Examples include the number of neighbors (k) in K-Nearest Neighbors, or the maximum depth allowed for a Decision Tree. Choosing good hyperparameters often requires experimentation — a process explored in depth in Module 7 (Grid Search, Random Search).

d) How They Interact

Hyperparameters influence how the training process happens, which in turn affects what parameter values the model ultimately learns. For example, choosing a different k value in KNN doesn't change how KNN itself works, but it changes what predictions the resulting model makes.

7. How It Works

1. Before training, you choose the algorithm's hyperparameters (e.g., how many neighbors KNN should consider).
2. During training (.fit()), the algorithm uses the training data to determine the best parameter values (e.g., the specific weights of a Linear Regression line).
3. The resulting trained model — hyperparameters plus learned parameters — is then used to make predictions on new data.

8. Real-World Example

Think of baking a cake. The oven temperature and baking time are like hyperparameters — you (the baker) set them before you start, based on experience or experimentation, and they aren't determined by the ingredients themselves. The actual chemical changes happening inside the batter as it bakes (rising, browning) are like parameters — they're determined automatically by the process, given the hyperparameters you chose.

9. Python Example

python
from sklearn.neighbors import KNeighborsClassifier

# Hyperparameter: n_neighbors is set manually, BEFORE training
model = KNeighborsClassifier(n_neighbors=3)

# Training data
X = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]

# Training - the model learns from the data using the chosen hyperparameter
model.fit(X, y)

prediction = model.predict([[9]])
print("Prediction:", prediction[0])

# Hyperparameters can be inspected directly
print("Hyperparameter n_neighbors:", model.n_neighbors)

Expected Output:

text
Prediction: 1
Hyperparameter n_neighbors: 3

10. Code Explanation

• n_neighbors=3 is a hyperparameter — WE chose this value manually before training even started; it is not learned from the data.
• model.fit(X, y) is where the actual learning happens — for KNN specifically, "training" mostly involves storing the data in a way that allows quick distance-based comparisons (KNN is explored fully in Module 4).
• model.predict([[9]]) uses the chosen hyperparameter (n_neighbors=3) to look at the 3 nearest training points and predict based on their labels.
• Changing n_neighbors to a different value (e.g., 1 or 5) would change the model's predictions — illustrating exactly how hyperparameters influence a model's behavior.

11. Advantages

• Separating parameters (learned) from hyperparameters (chosen) gives developers meaningful control over the training process.
• Tuning hyperparameters (Module 7) can significantly improve a model's performance without changing the underlying algorithm.

12. Limitations

• Poorly chosen hyperparameters can lead to a poorly performing model, even with a fundamentally sound algorithm.
• Searching for the best hyperparameters (Grid Search, Random Search) can be computationally expensive, especially with many hyperparameters to tune.

13. Common Mistakes

• Confusing parameters (learned automatically) with hyperparameters (set manually) — this is one of the most frequently tested distinctions in ML interviews.
• Assuming default hyperparameter values are always optimal for every dataset — they often need tuning.
• Forgetting that changing a hyperparameter requires retraining the model, since it affects the learning process itself.

14. Best Practices

• Always understand which hyperparameters a given algorithm exposes, and what each one controls.
• Use systematic approaches (Grid Search or Random Search, covered in Module 7) rather than guessing hyperparameter values randomly.
• Keep track of which hyperparameter values were used for each experiment, to enable fair comparisons.

15. Real-World Applications

• Tuning the number of neighbors in KNN-based recommendation systems.
• Tuning the maximum depth of Decision Trees and Random Forests to balance accuracy and overfitting.
• Tuning learning rates and network architecture in Deep Learning models (Module 10).

16. Interview-Oriented Points

• Be ready to clearly distinguish parameters (learned during training) from hyperparameters (set before training).
• Be able to give a concrete example of a hyperparameter for at least one algorithm (e.g., k in KNN, max_depth in Decision Trees).
• Understand why hyperparameter tuning is an important part of the ML workflow.

17. Exam-Oriented Points

• A model is the trained system used to make predictions.
• Parameters are learned automatically from data during training (e.g., weights in Linear Regression).
• Hyperparameters are set manually before training and control the learning process (e.g., k in KNN).

18. Comparison Table — Parameters vs Hyperparameters

19. Quick Revision

• A model is the complete trained system used to make predictions.
• Parameters are learned automatically from data during training (e.g., weights, slope).
• Hyperparameters are set manually before training and control how the learning process happens (e.g., k, max_depth).
• Choosing good hyperparameters (hyperparameter tuning) is a key skill covered in depth in Module 7.