Handling Missing Values

1. Introduction

Missing values are one of the most common problems in real-world datasets — a sensor might fail, a survey respondent might skip a question, or data might simply not have been recorded. This topic covers how to detect missing values and the main strategies for handling them, an essential preprocessing step before training any ML model.

2. What Does "Handling Missing Values" Mean?

Simple definition: Handling missing values means detecting where data is absent in a dataset and deciding how to deal with it — either by removing the affected rows/columns or by filling in reasonable replacement values.

Technical explanation: Missing values (often represented as NaN in Pandas) must be addressed before most ML algorithms can process a dataset, since most algorithms cannot handle undefined values directly; common strategies include deletion (dropping rows/columns) and imputation (filling in estimated values based on statistics or other methods).

3. Why is it Important?

• Most ML algorithms will raise errors or behave unpredictably if fed missing values directly.
• How you handle missing data can significantly affect model accuracy and reliability.
• Missing values are extremely common in real-world datasets — this is a skill you'll use in nearly every project.

4. Prerequisites

Comfort with Pandas (Module 1, Topic 6) and Basic Statistics (Module 1, Topic 8), since imputation often relies on mean/median/mode.

5. Core Concepts

1. Detecting missing values (isnull(), isna())
2. Deletion strategies (dropping rows or columns)
3. Imputation strategies (mean, median, mode, forward/backward fill)
4. Using Scikit-learn's SimpleImputer
5. Choosing the right strategy for a given situation

6. Detailed Explanation

a) Detecting Missing Values

df.isnull() (or the identical df.isna()) returns True/False for each cell, indicating whether it's missing. Combined with .sum(), it gives a count of missing values per column.

b) Deletion Strategies

• df.dropna() removes rows containing any missing values.
• df.dropna(axis=1) removes entire columns containing any missing values.
• Deletion is simple but can discard valuable data, especially if missing values are common.

c) Imputation Strategies

• Mean/Median Imputation: Replace missing numeric values with the column's mean or median — median is often preferred when outliers are present.
• Mode Imputation: Replace missing categorical values with the most frequent category.
• Forward/Backward Fill: Replace missing values using the previous (ffill) or next (bfill) valid value — useful for time-series data.

d) Using Scikit-learn's SimpleImputer

SimpleImputer provides a standardized, reusable way to perform imputation, which integrates cleanly into ML pipelines (fitting on training data, then applying consistently to test data).

e) Choosing the Right Strategy

The right approach depends on how much data is missing, why it's missing, and the nature of the column (numeric vs categorical). Dropping is reasonable when missing values are rare; imputation is preferred when dropping would discard too much valuable data.

7. How It Works

1. Detect missing values using isnull().sum() to understand the scope of the problem.
2. Decide whether to drop or impute, based on how much data is missing and its importance.
3. If imputing, choose an appropriate strategy (mean, median, mode, or fill method) based on the column type and distribution.
4. Apply the chosen method and verify that missing values have been properly handled (re-run isnull().sum()).

8. Real-World Example

In a medical dataset, a "blood pressure" reading might be missing for some patients due to equipment issues. Simply dropping every patient with a missing blood pressure reading could discard a large, valuable portion of the dataset. Instead, filling the missing values with the median blood pressure (a value unaffected by extreme outliers) preserves the dataset's size while still allowing the analysis to proceed reasonably.

9. Python Example

python
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

data = {
    "age": [25, np.nan, 30, 22, np.nan],
    "salary": [50000, 60000, np.nan, 45000, 52000],
    "city": ["Delhi", "Mumbai", np.nan, "Delhi", "Pune"]
}
df = pd.DataFrame(data)

print("Missing values per column:")
print(df.isnull().sum())

# Imputing numeric columns with mean
mean_imputer = SimpleImputer(strategy="mean")
df[["age", "salary"]] = mean_imputer.fit_transform(df[["age", "salary"]])

# Imputing categorical column with the most frequent value
mode_imputer = SimpleImputer(strategy="most_frequent")
df[["city"]] = mode_imputer.fit_transform(df[["city"]])

print("\nAfter imputation:")
print(df)
print("\nMissing values per column after imputation:")
print(df.isnull().sum())

Expected Output (approximate):

text
Missing values per column:
age       2
salary    1
city      1
dtype: int64

After imputation:
     age   salary    city
0  25.00  50000.0   Delhi
1  25.67  60000.0  Mumbai
2  30.00  51750.0   Delhi
3  22.00  45000.0   Delhi
4  25.67  52000.0    Pune

Missing values per column after imputation:
age       0
salary    0
city      0
dtype: int64

10. Code Explanation

• df.isnull().sum() counts missing values per column, giving a clear starting picture of the problem.
• SimpleImputer(strategy="mean") creates an imputer that will replace missing numeric values with each column's mean.
• .fit_transform(df[["age", "salary"]]) calculates the mean of each column from the available data and immediately fills in the missing values.
• SimpleImputer(strategy="most_frequent") is used for the categorical "city" column, filling missing entries with the most common city.
• Re-checking isnull().sum() afterward confirms that all missing values have been successfully handled.

11. Advantages

• Prevents ML algorithms from failing due to undefined (missing) values.
• Imputation preserves valuable data that would otherwise be discarded by deletion.
• SimpleImputer integrates cleanly into reusable, consistent ML pipelines.

12. Limitations

• Imputation introduces estimated (not real) values, which can slightly bias results, especially if a large proportion of data is missing.
• Deletion can discard valuable information, especially with small datasets or high proportions of missing data.
• Choosing the wrong strategy (e.g., mean imputation with heavy outliers present) can distort the data's true distribution.

13. Common Mistakes

• Dropping rows/columns without first checking how much data would actually be lost.
• Using mean imputation on data with significant outliers, when median would be more robust.
• Forgetting to check for missing values at all before training a model, leading to confusing errors.
• Applying different imputation values to training and test sets inconsistently (the imputer should be fit only on training data).

14. Best Practices

• Always start by checking isnull().sum() to understand the scope of missing data.
• Prefer median over mean for imputation when a column has outliers.
• Fit imputers only on training data, then apply the same fitted transformation to test data, to avoid data leakage.
• Consider whether "missingness" itself might carry meaningful information (e.g., a missing survey answer might indicate reluctance to disclose) before blindly filling it.

15. Real-World Applications

• Handling missing sensor readings in IoT and industrial datasets.
• Filling in missing survey responses in social science and market research data.
• Managing incomplete medical records in healthcare analytics.

16. Interview-Oriented Points

• Be ready to explain the difference between deletion and imputation, and when each is appropriate.
• Understand why median is often preferred over mean for imputation in the presence of outliers.
• Be able to explain why an imputer should be fit on training data only, not on the full dataset (to avoid data leakage into the test set).

17. Exam-Oriented Points

• df.isnull().sum() detects and counts missing values per column.
• dropna() removes rows/columns with missing values; imputation fills them in instead.
• Common imputation strategies: mean, median (numeric), mode/most frequent (categorical), forward/backward fill (time series).
• SimpleImputer from Scikit-learn provides a standardized, reusable imputation tool.

18. Comparison Table — Dropping vs Imputing Missing Values

19. Quick Revision

• df.isnull().sum() detects and counts missing values per column.
• Deletion (dropna()) removes affected rows/columns; imputation fills in estimated values instead.
• Numeric columns are often imputed with mean or median; categorical columns with mode (most frequent value).
• SimpleImputer from Scikit-learn provides a standardized, pipeline-friendly imputation tool.
• Always fit imputers on training data only, to avoid data leakage into the test set.