PCA (Principal Component Analysis)

1. Introduction

This final topic of Module 5 shifts from clustering to the second major category of Unsupervised Learning: dimensionality reduction. PCA (Principal Component Analysis) is the most widely used dimensionality reduction technique, and it directly builds on the Linear Algebra concepts (vectors, matrices) introduced back in Module 1.

2. What is PCA?

Simple definition: PCA is an unsupervised technique that reduces the number of features in a dataset while preserving as much of the original information (variance) as possible.

Technical explanation: PCA transforms a dataset's original correlated features into a smaller set of new, uncorrelated features called principal components, ordered by how much variance (information) they capture from the original data, allowing dimensionality reduction with minimal information loss.

3. Why is it Important?

• High-dimensional data (many features) can be difficult to visualize, slow to train models on, and prone to the "curse of dimensionality."
• PCA allows visualizing complex, high-dimensional data in 2D or 3D by reducing it to just a few principal components.
• It's widely used as a preprocessing step to speed up training and reduce noise before applying other ML algorithms.

4. Prerequisites

Comfort with Linear Algebra Basics (Module 1, Topic 10) and Feature Selection (Module 3, Topic 8), since PCA is another (different) approach to reducing feature count.

5. Core Concepts

1. Variance and information
2. Principal components
3. Explained variance ratio
4. Choosing the number of components to keep

6. Detailed Explanation

a) Variance and Information

In PCA, "variance" is treated as a proxy for "information" — a feature (or combination of features) that varies a lot across data points is considered to carry more useful information than one that barely changes at all.

b) Principal Components

Principal components are new, artificially constructed features — each one is a specific weighted combination of the original features, calculated so that: (1) the first principal component captures the maximum possible variance in the data, (2) the second principal component captures the maximum remaining variance while being completely uncorrelated with the first, and so on.

c) Explained Variance Ratio

Each principal component "explains" a certain percentage of the original dataset's total variance. The explained variance ratio tells you how much information (variance) each component captures — the first component typically explains the most, with each subsequent component explaining progressively less.

d) Choosing the Number of Components

You choose how many principal components to keep based on how much cumulative explained variance you're comfortable retaining (e.g., keeping enough components to explain 95% of the original variance), balancing dimensionality reduction against information loss.

7. How It Works

1. Standardize the data (PCA is sensitive to feature scale, similar to KNN and SVM — Module 3, Topic 7).
2. Calculate the principal components — new axes that capture the maximum variance in the data, ordered from most to least variance explained.
3. Decide how many components to keep, based on cumulative explained variance.
4. Transform the original data onto these selected principal components, reducing the number of features while preserving most of the original information.

8. Real-World Example

Imagine a dataset describing students with 20 different exam scores. Many of these scores might be highly correlated (a student who does well in Algebra likely also does well in Calculus). PCA could combine these 20 correlated features into just 2 or 3 new "principal components" — perhaps one representing overall "quantitative ability" and another representing "verbal ability" — capturing most of the original information in far fewer dimensions, making the data much easier to visualize and analyze.

9. Mathematical Explanation

PCA's underlying mathematics involves eigenvectors and eigenvalues of the data's covariance matrix (a more advanced linear algebra topic) — conceptually:

Explained Variance Ratio (Conceptual Formula):

Explained Variance Ratio for component i = eigenvalueᵢ / Σ(all eigenvalues)

Where:

• eigenvalueᵢ = the variance captured by the i-th principal component
• The denominator sums the variance captured across ALL components (representing the total original variance)

Numerical Example:

Suppose a dataset's PCA produces these eigenvalues (variance captured) for its components: [8, 3, 1]

• Total variance = 8 + 3 + 1 = 12
• Explained variance ratio for Component 1 = 8/12 ≈ 0.667 (66.7%)
• Explained variance ratio for Component 2 = 3/12 = 0.25 (25%)
• Explained variance ratio for Component 3 = 1/12 ≈ 0.083 (8.3%)

Interpreting the Result: Keeping just the first two components would preserve 66.7% + 25% = 91.7% of the original data's total variance/information, while reducing the dataset from 3 dimensions down to just 2 — a meaningful reduction with relatively modest information loss.

10. Python Example

python
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

# A dataset with 4 features (some likely correlated)
X = np.array([
    [90, 85, 88, 92],
    [70, 65, 68, 72],
    [60, 55, 58, 62],
    [95, 92, 94, 96],
    [50, 48, 52, 55]
])

# Standardizing before PCA (important, since PCA is scale-sensitive)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X_scaled)

print("Original shape:", X.shape)
print("Reduced shape:", X_reduced.shape)
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Total variance explained:", sum(pca.explained_variance_ratio_))

Expected Output (approximate):

text
Original shape: (5, 4)
Reduced shape: (5, 2)
Explained variance ratio: [0.94 0.04]
Total variance explained: 0.98

11. Code Explanation

• StandardScaler().fit_transform(X) scales the data first, since PCA (like KNN and SVM) is sensitive to feature scale.
• PCA(n_components=2) specifies that we want to reduce the data down to just 2 principal components.
• pca.fit_transform(X_scaled) calculates the principal components and transforms the original 4-feature data into just 2 new features.
• pca.explained_variance_ratio_ shows that the first component alone captures about 94% of the original data's variance, with the second adding another 4% — together preserving about 98% of the original information despite cutting the number of features in half.

12. Advantages

• Reduces dimensionality while preserving most of the original data's important information/variance.
• Makes high-dimensional data easier to visualize (typically reduced to 2D or 3D).
• Can improve training speed and reduce noise for subsequent ML algorithms.

13. Limitations

• The resulting principal components are new, artificial combinations of original features, making them harder to directly interpret (unlike Feature Selection, which keeps original, meaningful features).
• PCA only captures linear relationships between features, potentially missing important non-linear structure.
• Sensitive to feature scale, requiring standardization beforehand.

14. Common Mistakes

• Applying PCA without first scaling the data, leading to features with larger ranges dominating the principal components.
• Assuming PCA components retain the same meaning/interpretability as the original features.
• Reducing dimensions too aggressively, losing significant information without checking the cumulative explained variance.

15. Best Practices

• Always scale/standardize data before applying PCA.
• Check the cumulative explained variance ratio to make an informed decision about how many components to keep.
• Remember that principal components are mathematical constructs, not directly interpretable original features — use Feature Selection instead when interpretability of original features matters.

16. Real-World Applications

• Visualizing high-dimensional datasets (like genetic data or image features) in 2D or 3D.
• Speeding up training of ML models on datasets with very many features.
• Noise reduction in datasets where less significant components mostly represent noise rather than genuine signal.

17. Interview-Oriented Points

• Be ready to explain what a "principal component" is and how it relates to variance.
• Understand the explained variance ratio and how it guides the choice of how many components to keep.
• Be able to explain the key difference between PCA (creates new artificial features) and Feature Selection (keeps original features).

18. Exam-Oriented Points

• PCA reduces dimensionality by creating new, uncorrelated principal components that capture maximum variance.
• The explained variance ratio shows how much information each component preserves.
• PCA requires feature scaling beforehand, since it's sensitive to the scale of original features.

19. Comparison Table — PCA vs Feature Selection

20. Quick Revision

• PCA reduces dimensionality by creating new principal components that capture maximum variance from the original data.
• The explained variance ratio shows how much information each component preserves, guiding how many to keep.
• PCA requires feature scaling beforehand and produces components that are harder to directly interpret than original features.
• Use PCA when dimensionality reduction itself is the goal; use Feature Selection when keeping original, interpretable features matters more.