Machine Learning

Machine Learning (ML) is what lets software improve at a task by learning patterns from data, instead of following only explicitly hand-written rules. This file starts with the fundamental concepts every ML practitioner needs, then walks through the core algorithms using Python's most popular ML library: Scikit-learn.

bash
pip install scikit-learn pandas numpy

1. AI vs ML vs Deep Learning

What is it?

These three terms are related but not identical — understanding the distinction is a very common interview question.

Definition: Artificial Intelligence is the broad field of making machines behave intelligently. Machine Learning is a subset of AI where machines learn patterns from data rather than following explicit rules. Deep Learning is a subset of ML using multi-layered neural networks.

Comparison Table

Explanation: Every Deep Learning system is Machine Learning, and every Machine Learning system is AI — but not every AI system uses Machine Learning (some AI is purely rule-based), and not every ML system uses Deep Learning (many use simpler, non-neural-network algorithms, covered in this file).

2. Supervised vs Unsupervised Learning

What is it?

• Supervised learning — the model learns from data that already has the "correct answers" (labels) attached, and learns to predict those labels for new data.
• Unsupervised learning — the model finds patterns or structure in data that has no labels at all.

Definition: Supervised learning trains a model using labeled data (inputs paired with known correct outputs). Unsupervised learning finds hidden patterns in unlabeled data.

Real-World Example

• Supervised: Given past house sales (size, location, actual sale price), predict the price of a new house. The "actual sale price" is the label the model learns from.
• Unsupervised: Given customer purchase data with no predefined categories, group customers into natural clusters based on similar buying behavior.

Comparison Table

3. Regression, Classification, and Clustering

What is it?

• Regression — predicting a continuous numeric value (e.g., a price, a temperature, a score).
• Classification — predicting a category/class (e.g., spam or not spam, pass or fail).
• Clustering — grouping similar data points together, without predefined categories (unsupervised).

Real-World Example

4. Features and Labels

What is it?

• Features (also called inputs, or X) — the measurable properties used to make a prediction.
• Label (also called the target, or y) — the actual answer the model is trying to predict.

Simple Example

Predicting a student's exam score based on hours studied:

Important Points

• Features are what you already know; the label is what you're trying to predict.
• In supervised learning, both features and labels are present in your training data.

5. Train/Test Split

What is it?

Before training a model, data is split into a training set (used to teach the model) and a testing set (kept completely separate, used only to check how well the model performs on data it has never seen).

Why do we use it?

If you test a model on the exact same data it was trained on, it could just be "memorizing" the answers rather than genuinely learning patterns — this gives a falsely optimistic sense of how well it actually works. Testing on unseen data reveals its real performance.

Simple Example

python
from sklearn.model_selection import train_test_split

X = [[5, 90], [2, 60], [8, 95], [4, 70], [9, 98], [1, 40]]   # features
y = [78, 45, 92, 65, 95, 30]                                    # labels

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(len(X_train), len(X_test))   # e.g. 4 2 (80% train, 20% test)

Explanation: test_size=0.2 reserves 20% of the data for testing. random_state=42 ensures the split is reproducible — running the code again gives the exact same split, useful for consistent comparisons.

Important Points

• A common split ratio is 80% training / 20% testing, though this can vary.
• Never evaluate a model's real-world performance using its own training data.

6. Model Evaluation

What is it?

After training, you need metrics to measure how good the model's predictions actually are.

For Regression

python
from sklearn.metrics import mean_squared_error, r2_score

mse = mean_squared_error(y_test, predictions)
r2 = r2_score(y_test, predictions)

For Classification

python
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix

accuracy = accuracy_score(y_test, predictions)
precision = precision_score(y_test, predictions)
recall = recall_score(y_test, predictions)

Real-World Example

For a disease-detection model, recall matters enormously — missing an actual sick patient (a "false negative") is far more dangerous than a false alarm. Accuracy alone can be misleading, especially when one class is much rarer than the other.

Important Points

• Accuracy alone can be misleading on imbalanced data (e.g., 99% "not spam" emails) — precision and recall give a fuller picture.
• Always choose evaluation metrics based on what actually matters for your specific real-world problem.

7. Cross-Validation

What is it?

Instead of a single train/test split, cross-validation splits the data into multiple "folds," training and testing multiple times on different portions — giving a more reliable estimate of model performance.

Simple Example — K-Fold Cross-Validation

python
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression

model = LinearRegression()
scores = cross_val_score(model, X, y, cv=5)   # 5-fold cross-validation

print(scores)
print("Average score:", scores.mean())

Explanation: cv=5 splits the data into 5 equal parts; the model trains on 4 parts and tests on the remaining 1, repeating this process 5 times (each part gets a turn as the test set), then averages the results — reducing the chance that one lucky/unlucky split skews your evaluation.

Important Points

• Cross-validation gives a more robust, reliable estimate of model performance than a single train/test split.
• Commonly used with 5 or 10 folds.

8. Feature Engineering (Brief Overview)

What is it?

Feature engineering means creating, transforming, or selecting the features fed into a model to improve its performance — often the single most impactful part of a real ML project.

Simple Example

python
import pandas as pd

df = pd.DataFrame({"birth_year": [2000, 1995, 2003]})
df["age"] = 2026 - df["birth_year"]   # engineered feature: age is often more useful than raw birth year

Important Points

• Good feature engineering often improves model performance more than switching to a fancier algorithm.
• Common techniques include scaling numeric features, encoding categorical variables (e.g., converting "Yes"/"No" into 1/0), and creating new, more meaningful features from existing ones.

9. Core Algorithms

9.1 Linear Regression (Regression)

What is it?

Predicts a continuous numeric value by fitting a straight line (or hyperplane, with multiple features) through the data.

python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

hours_studied = [[1], [2], [3], [4], [5], [6], [7], [8]]
exam_scores = [35, 45, 50, 60, 65, 75, 82, 90]

X_train, X_test, y_train, y_test = train_test_split(hours_studied, exam_scores, test_size=0.25, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(predictions)

new_prediction = model.predict([[9]])
print(f"Predicted score for 9 hours studied: {new_prediction[0]:.2f}")

Real-World Use: Predicting house prices, sales forecasts, exam scores based on study hours.

9.2 Logistic Regression (Classification)

What is it?

Despite the name "regression," this is actually a classification algorithm — predicting a category (typically yes/no, or pass/fail), based on a probability.

python
from sklearn.linear_model import LogisticRegression

hours_studied = [[1], [2], [3], [4], [5], [6], [7], [8]]
passed = [0, 0, 0, 1, 1, 1, 1, 1]   # 0 = fail, 1 = pass

model = LogisticRegression()
model.fit(hours_studied, passed)

prediction = model.predict([[3.5]])
print("Pass" if prediction[0] == 1 else "Fail")

Real-World Use: Spam detection, disease diagnosis (positive/negative), customer churn prediction (will leave / will stay).

9.3 Decision Tree (Classification or Regression)

What is it?

A model that makes decisions by asking a series of yes/no questions about the features, forming a tree-like structure of splits.

python
from sklearn.tree import DecisionTreeClassifier

features = [[25, 50000], [45, 80000], [35, 60000], [22, 30000], [50, 90000]]
buys_product = [0, 1, 0, 0, 1]

model = DecisionTreeClassifier()
model.fit(features, buys_product)

prediction = model.predict([[30, 55000]])
print(prediction)

Real-World Use: Loan approval decisions, medical diagnosis systems — valued for being relatively easy to interpret and explain to non-technical people.

9.4 Random Forest (Classification or Regression)

What is it?

An "ensemble" method that builds many decision trees and combines their predictions (usually by majority vote) — generally more accurate and less prone to overfitting than a single decision tree.

python
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100)
model.fit(features, buys_product)

prediction = model.predict([[30, 55000]])
print(prediction)

Explanation: n_estimators=100 builds 100 individual decision trees, each trained slightly differently, then combines their votes for a more robust final prediction.

Real-World Use: Widely used in production ML systems for its strong accuracy and resistance to overfitting compared to a single tree.

9.5 K-Nearest Neighbors — KNN (Classification or Regression)

What is it?

Classifies a new data point based on the majority class among its "k" closest neighbors in the training data — no real "training" happens beyond storing the data.

python
from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier(n_neighbors=3)
model.fit(features, buys_product)

prediction = model.predict([[30, 55000]])
print(prediction)

Explanation: n_neighbors=3 means the prediction is based on the 3 closest known data points to the new one — whichever class is most common among those 3 "neighbors" wins.

Real-World Use: Recommendation systems, simple classification problems with clearly separable groups.

9.6 Support Vector Machine — SVM (Classification)

What is it?

Finds the best possible boundary (a "hyperplane") that separates classes with the maximum possible margin between them.

python
from sklearn.svm import SVC

model = SVC()
model.fit(features, buys_product)

prediction = model.predict([[30, 55000]])
print(prediction)

Real-World Use: Text classification, image classification, particularly effective in high-dimensional feature spaces.

9.7 Naive Bayes (Classification)

What is it?

A probability-based classifier built on Bayes' Theorem, assuming features are independent of each other (the "naive" assumption) — surprisingly effective in practice, especially for text.

python
from sklearn.naive_bayes import GaussianNB

model = GaussianNB()
model.fit(features, buys_product)

prediction = model.predict([[30, 55000]])
print(prediction)

Real-World Use: Spam email filtering, sentiment analysis — extremely fast to train, even on large datasets.

9.8 K-Means (Clustering — Unsupervised)

What is it?

Groups data into k clusters based on similarity, with no labels required at all — the algorithm discovers the groupings entirely on its own.

python
from sklearn.cluster import KMeans

customer_data = [[20, 15000], [22, 18000], [45, 80000], [48, 85000], [50, 90000], [25, 20000]]

model = KMeans(n_clusters=2, random_state=42, n_init=10)
model.fit(customer_data)

print(model.labels_)         # which cluster each data point belongs to
print(model.cluster_centers_)   # the "center" of each cluster

Explanation: n_clusters=2 tells K-Means to find exactly 2 groups. .labels_ shows which cluster (0 or 1) each original data point was assigned to.

Real-World Use: Customer segmentation, grouping similar products, image compression.

Comparison Table — Choosing an Algorithm

Common Beginner Mistakes — Summary for This Section

• Evaluating a model on the same data it was trained on, giving a falsely optimistic result.
• Relying solely on accuracy for classification problems with imbalanced classes.
• Confusing Logistic Regression (classification) with Linear Regression (regression) due to the similar name.
• Skipping feature engineering and jumping straight to trying fancier algorithms.

Cheat Sheet — Machine Learning

python
from sklearn.model_selection import train_test_split, cross_val_score
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Regression
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

# Classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB

# Clustering
from sklearn.cluster import KMeans

# Evaluation
from sklearn.metrics import accuracy_score, mean_squared_error, r2_score

Mini Project: Student Performance Prediction

Objective

Build a model that predicts whether a student will pass or fail based on hours studied and attendance percentage, using Logistic Regression.

Requirements

• Use sample training data (hours studied, attendance, pass/fail).
• Train a classification model.
• Evaluate its accuracy and predict a new student's outcome.

Concepts Used

Features/labels, train/test split, Logistic Regression, model evaluation.

Complete Code

python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Features: [hours_studied, attendance_percent]
X = [
    [1, 50], [2, 55], [3, 60], [4, 65], [5, 70],
    [6, 75], [7, 85], [8, 90], [9, 95], [10, 98]
]
# Labels: 0 = Fail, 1 = Pass
y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f"Model Accuracy: {accuracy * 100:.2f}%")

new_student = [[4.5, 68]]
result = model.predict(new_student)
print("Prediction for new student:", "Pass" if result[0] == 1 else "Fail")

probability = model.predict_proba(new_student)
print(f"Confidence: Fail={probability[0][0]:.2f}, Pass={probability[0][1]:.2f}")

Code Explanation

• X holds two features per student (hours studied, attendance); y holds the known pass/fail outcome for each.
• After training, accuracy_score checks how many of the held-out test predictions matched the actual known outcomes.
• .predict_proba() shows the model's confidence in each class — useful when the decision is close, rather than just a hard yes/no.

Sample Output

Model Accuracy: 100.00%
Prediction for new student: Pass
Confidence: Fail=0.18, Pass=0.82

Possible Improvements

• Add more features (assignment scores, class participation).
• Use a larger, real dataset instead of small hardcoded sample data.
• Compare Logistic Regression's accuracy against a Decision Tree or Random Forest on the same data.

Challenge Task

Extend the project to predict an actual numeric exam score (using Linear Regression) instead of just pass/fail, and compare the two approaches.

Interview Questions

Q1. What is the difference between AI, ML, and Deep Learning? Answer: AI is the broad field of making machines act intelligently. ML is a subset of AI focused on learning patterns from data. Deep Learning is a further subset of ML using multi-layered neural networks.

Q2. What is the difference between supervised and unsupervised learning? Answer: Supervised learning trains on labeled data (with known correct answers) to predict outcomes for new data. Unsupervised learning finds patterns or groupings in data that has no labels at all.

Q3. What is the difference between regression and classification? Answer: Regression predicts a continuous numeric value. Classification predicts a discrete category or class.

Q4. Why is Logistic Regression used for classification despite its name? Answer: It calculates a probability (using a logistic/sigmoid function) that's then used to assign a class label (e.g., above 0.5 probability = one class, below = the other) — the "regression" refers to the underlying calculation, not the type of output.

Q5. Why is a train/test split necessary? Answer: To evaluate how well a model performs on data it has never seen, avoiding a falsely optimistic result from testing on the same data used for training.

Q6. What is the difference between precision and recall? Answer: Precision measures how many of the model's positive predictions were actually correct. Recall measures how many of the actual positive cases the model successfully identified.

Q7. What is the difference between a Decision Tree and a Random Forest? Answer: A Decision Tree is a single tree-based model. A Random Forest builds many decision trees and combines their predictions, generally improving accuracy and reducing overfitting compared to any single tree.

Q8. What is K-Means used for, and is it supervised or unsupervised? Answer: K-Means is a clustering algorithm, used to group similar data points together. It's unsupervised — it requires no labeled data at all.

Practice Questions

Beginner

1. Split a small dataset of 10 samples into training and testing sets using train_test_split.
2. Train a Linear Regression model to predict a value based on a single feature, and print its predictions on test data.
3. Train a Logistic Regression model on a simple pass/fail dataset and calculate its accuracy.
4. Explain, in your own words, the difference between features and labels.
5. Train a K-Means model with 2 clusters on a small sample dataset and print each point's assigned cluster.

Intermediate

1. Train and compare a Decision Tree and a Random Forest on the same classification dataset, comparing their accuracy.
2. Use cross-validation (cross_val_score) to evaluate a Linear Regression model on a small dataset.
3. Train a KNN classifier with different values of k (e.g., 1, 3, 5) and observe how the predictions change.
4. Calculate precision and recall (not just accuracy) for a classification model on an imbalanced sample dataset.
5. Engineer a new feature (e.g., "study hours per day" from "total study hours" and "days until exam") and use it in a model.

Challenge

1. Build a complete ML pipeline: load a small dataset, split it, engineer at least one new feature, train a Random Forest, and evaluate it with accuracy, precision, and recall.
2. Compare the performance of Logistic Regression, Decision Tree, and SVM on the same classification dataset, and explain which performed best and why.
3. Extend the Student Performance Prediction mini project to use K-Means clustering to group students into "at-risk," "average," and "high-performing" categories based on their features, without using the pass/fail labels at all.