Training, Validation & Testing

1. Introduction

A single dataset used carelessly can give you a very misleading impression of how good your model actually is. To build reliable ML models, we split our data into three distinct parts, each with a very specific purpose: training, validation, and testing. Understanding this split is essential before you build your first real model in Module 4.

2. What are Training, Validation, and Testing?

Simple definition: Training, validation, and testing are three separate portions of a dataset, each used at a different stage of building and evaluating an ML model — training to teach the model, validation to tune it, and testing to give a final, honest assessment of its performance.

Technical explanation: The training set is used to fit model parameters; the validation set is used during development to tune hyperparameters and make model-selection decisions; the test set is held out entirely until the very end, providing an unbiased estimate of the model's performance on genuinely unseen data.

3. Why is it Important?

• Without this split, you risk overestimating your model's real-world performance — a model can appear excellent simply because it memorized the data it was evaluated on.
• Proper splitting is essential for detecting and preventing overfitting (explored in depth in Module 7).
• Nearly every ML project and interview question assumes familiarity with this fundamental workflow step.

4. Prerequisites

Comfort with the general ML workflow described in Topic 2 (What is Machine Learning?).

5. Core Concepts

1. Training set
2. Validation set
3. Test set
4. Typical split ratios
5. Why the test set must remain untouched until the very end

6. Detailed Explanation

a) Training Set

The training set is the portion of data the model actually learns from — its parameters are adjusted based on this data during the .fit() process.

b) Validation Set

The validation set is used during development to make decisions — such as comparing different models, or tuning hyperparameters (explored further in Module 7) — without touching the final test set.

c) Test Set

The test set is kept completely separate and untouched until the very end of the project. It's used exactly once, to give a final, honest, unbiased measurement of how well the model is likely to perform on truly new, real-world data.

d) Typical Split Ratios

Common splits include 70% training / 15% validation / 15% testing, or 80% training / 10% validation / 10% testing — though exact ratios can vary depending on dataset size and project needs. For smaller datasets, techniques like Cross-Validation (covered in Module 6) are often used instead of a single fixed validation split.

e) Why the Test Set Must Stay Untouched

If you repeatedly check your model's performance on the test set and make adjustments based on those results, you're effectively "leaking" information from the test set into your model-building process — making your final performance estimate misleadingly optimistic. This is why the validation set exists: to allow experimentation without contaminating the test set.

7. How It Works

1. Split your full dataset into training, validation, and test sets (commonly using a function like train_test_split).
2. Train your model using only the training set.
3. Evaluate different models or hyperparameter settings using the validation set, adjusting as needed.
4. Once you've finalized your model, evaluate it exactly once on the test set to get your final performance estimate.

8. Real-World Example

Think of studying for an exam using a set of practice questions with answers (training set), then taking a few practice tests to check your understanding and identify weak areas (validation set), and finally sitting the real, unseen final exam just once (test set). If you somehow saw the real exam questions beforehand and studied them directly, your exam score would no longer honestly reflect your true understanding — this is exactly the problem that keeping the test set untouched prevents.

9. Python Example

python
from sklearn.model_selection import train_test_split
import numpy as np

# Sample dataset: 10 examples with 1 feature each
X = np.array([[i] for i in range(1, 11)])
y = np.array([i * 2 for i in range(1, 11)])

# First split: separate out the test set (e.g., 20% of data)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Second split: separate remaining data into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, random_state=42
)

print("Training set size:", len(X_train))
print("Validation set size:", len(X_val))
print("Test set size:", len(X_test))

Expected Output:

text
Training set size: 6
Validation set size: 2
Test set size: 2

10. Code Explanation

• train_test_split(X, y, test_size=0.2, ...) first separates out 20% of the data as the test set, leaving 80% for training and validation combined.
• The second train_test_split() call further divides that remaining 80% into training and validation portions (here, 25% of the remaining data becomes validation, which works out to 20% of the original full dataset).
• random_state=42 ensures the split is reproducible — running the code again gives the exact same split.
• Notice the test set (X_test, y_test) is created once and should not be touched again until final evaluation.

11. Advantages

• Provides an honest, unbiased estimate of real-world model performance.
• Helps detect overfitting early, before a model is deployed.
• Establishes a disciplined, repeatable workflow for model development.

12. Limitations

• Splitting reduces the amount of data available for actual training, which can be a concern with small datasets.
• A single random split can sometimes be unrepresentative by chance, especially with small or imbalanced datasets (Cross-Validation, covered in Module 6, helps address this).

13. Common Mistakes

• Evaluating a model on the test set multiple times during development and adjusting based on those results — this defeats its purpose entirely.
• Skipping a validation set entirely and instead tuning hyperparameters directly against the test set.
• Using an inappropriate split ratio for very small datasets, leaving too little data for reliable training.

14. Best Practices

• Always separate a test set early, and don't touch it again until the very end of your project.
• Use the validation set (or Cross-Validation) for all experimentation and hyperparameter tuning.
• Set a random_state for reproducibility when splitting data.
• For very small datasets, prefer Cross-Validation over a single fixed validation split (covered in Module 6).

15. Real-World Applications

• Standard practice in virtually every real-world ML project, from house price prediction to fraud detection.
• Used in Kaggle competitions and academic research to ensure fair, honest model comparisons.
• A required step before deploying any ML model into production.

16. Interview-Oriented Points

• Be ready to explain the distinct purpose of each of the three sets: training, validation, and testing.
• Understand why the test set must remain untouched until final evaluation.
• Be able to explain what "data leakage" means in this context and why it's a serious problem.

17. Exam-Oriented Points

• Training set: used to fit the model's parameters.
• Validation set: used to tune hyperparameters and compare models during development.
• Test set: used exactly once, at the end, for a final unbiased performance estimate.

18. Comparison Table — Training vs Validation vs Testing

19. Quick Revision

• Training set teaches the model; validation set tunes it; test set gives a final, honest performance check.
• The test set must remain untouched until the very end to avoid misleadingly optimistic results.
• train_test_split() from Scikit-learn is the standard tool for creating these splits.
• Common split ratios: roughly 70–80% training, with the remainder divided between validation and testing.