Deep Learning

Deep Learning is the branch of Machine Learning behind some of the most impressive recent AI breakthroughs — image recognition, speech recognition, and large language models. It's built on neural networks, loosely inspired by how neurons in the brain connect and pass signals to each other.

bash
pip install tensorflow torch

1. What is a Neural Network?

What is it?

A neural network is a model made of layers of connected "neurons," each performing a small calculation, working together to learn complex patterns from data.

Definition: A neural network is a computational model composed of layers of interconnected nodes (neurons), designed to recognize patterns in data.

Structure of a Neural Network

How a Single Neuron Works

Each neuron performs a simple calculation:

output = activation_function((input1 × weight1) + (input2 × weight2) + ... + bias)

Explanation: Every connection between neurons has a weight (how important that input is) and every neuron has a bias (an adjustable offset). During training, the network gradually adjusts these weights and biases to make its predictions more accurate.

Why "Deep" Learning?

"Deep" refers to having many hidden layers stacked together — each layer builds on patterns detected by the previous one (e.g., in image recognition: edges → shapes → objects → complete scenes).

Real-World Example

A neural network trained to recognize handwritten digits (0-9) takes pixel values as input, and after passing through several hidden layers, outputs which digit (0-9) it thinks the image represents.

2. Tensors

What is it?

A tensor is simply a multi-dimensional array — the fundamental data structure used throughout deep learning, similar in spirit to NumPy arrays (covered in the Data Analysis file), but with additional capabilities needed for deep learning (like automatic gradient tracking).

Examples of Tensor Dimensions

Important Points

• Both PyTorch and TensorFlow use tensors as their core data structure.
• Tensors can run on a GPU (graphics card) for dramatically faster computation than a regular CPU — essential for training large models efficiently.

3. Activation Functions

What is it?

An activation function decides whether (and how strongly) a neuron "fires," introducing non-linearity into the network — without this, a neural network (no matter how many layers) would behave just like a single simple linear equation, unable to learn complex patterns.

Common Activation Functions

Simple Example (Conceptual)

python
def relu(x):
    return max(0, x)

print(relu(5))    # 5
print(relu(-3))   # 0

Important Points

• ReLU is the most widely used activation function for hidden layers today, due to its simplicity and effectiveness.
• The output layer's activation function depends on the task: Sigmoid for yes/no, Softmax for multiple categories.

4. Loss Functions

What is it?

A loss function measures how wrong the model's predictions are compared to the actual correct answers — the number the network tries to minimize during training.

Common Loss Functions

Important Points

• A lower loss value means the model's predictions are closer to the correct answers.
• The choice of loss function should match the type of problem (regression vs classification).

5. Optimizers

What is it?

An optimizer is the algorithm that adjusts the network's weights and biases to reduce the loss, based on the gradients calculated during backpropagation (see below).

Common Optimizers

Important Points

• Adam is the most commonly used optimizer in modern deep learning, thanks to its strong general performance with minimal tuning.

6. Backpropagation (Conceptual Overview)

What is it?

Backpropagation is the algorithm that lets a neural network learn — it calculates exactly how much each weight contributed to the final error, then adjusts every weight slightly to reduce that error, working backward from the output layer to the input layer.

Definition: Backpropagation is the algorithm used to calculate how much each weight in a neural network contributed to the prediction error, allowing the optimizer to adjust weights accordingly.

How It Works (Simplified)

1. Forward pass — input data flows through the network, producing a prediction.
2. Calculate loss — compare the prediction to the actual correct answer.
3. Backward pass (backpropagation) — calculate how much each weight contributed to the error.
4. Update weights — the optimizer nudges each weight slightly to reduce the error.
5. Repeat this cycle many times (over many "epochs") until the loss becomes acceptably small.

Important Points

• You rarely implement backpropagation manually — frameworks like PyTorch and TensorFlow handle it automatically ("autograd").
• Understanding the concept (even without deriving the math) is valuable for understanding why training takes many repeated passes over the data.

7. Building a Neural Network with TensorFlow/Keras

What is it?

Keras is a high-level, beginner-friendly API built into TensorFlow — one of the easiest ways to build and train neural networks in Python.

Simple Example — Classifying Pass/Fail

python
import tensorflow as tf
from tensorflow import keras
import numpy as np

# Sample data: [hours_studied, attendance_percent]
X = np.array([[1, 50], [2, 55], [3, 60], [4, 65], [5, 70],
              [6, 75], [7, 85], [8, 90], [9, 95], [10, 98]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])   # 0 = Fail, 1 = Pass

model = keras.Sequential([
    keras.layers.Dense(8, activation="relu", input_shape=(2,)),
    keras.layers.Dense(4, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid")
])

model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

model.fit(X, y, epochs=50, verbose=0)

prediction = model.predict(np.array([[4.5, 68]]))
print("Pass probability:", prediction[0][0])

Explanation of the Code

• keras.Sequential([...]) stacks layers in order — data flows straight through from the first to the last.
• Dense(8, activation="relu") creates a fully-connected hidden layer with 8 neurons.
• The final Dense(1, activation="sigmoid") outputs a single value between 0 and 1 — perfect for binary (pass/fail) classification.
• model.compile() configures the loss function and optimizer before training begins.
• model.fit(X, y, epochs=50) trains the network for 50 full passes over the data.

Important Points

• Keras dramatically simplifies building neural networks compared to writing the underlying math manually.
• epochs controls how many times the model sees the entire training dataset during training.

8. Building a Neural Network with PyTorch

What is it?

PyTorch is another major deep learning framework, popular especially in research, offering more manual, flexible control over the training process compared to Keras's higher-level abstraction.

Simple Example

python
import torch
import torch.nn as nn

X = torch.tensor([[1, 50], [2, 55], [3, 60], [4, 65], [5, 70],
                   [6, 75], [7, 85], [8, 90], [9, 95], [10, 98]], dtype=torch.float32)
y = torch.tensor([[0], [0], [0], [0], [1], [1], [1], [1], [1], [1]], dtype=torch.float32)

class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 8)
        self.layer2 = nn.Linear(8, 1)
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.sigmoid(self.layer2(x))
        return x

model = SimpleNet()
loss_function = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(100):
    predictions = model(X)
    loss = loss_function(predictions, y)

    optimizer.zero_grad()
    loss.backward()      # backpropagation happens here, automatically
    optimizer.step()      # update the weights

new_student = torch.tensor([[4.5, 68]], dtype=torch.float32)
print("Pass probability:", model(new_student).item())

Explanation of the Code

• PyTorch defines a network as a Python class inheriting from nn.Module, with layers set up in __init__ and the data flow defined in forward().
• loss.backward() automatically calculates every weight's contribution to the error (backpropagation) — you never derive this math by hand.
• optimizer.step() applies the actual weight updates, based on those calculated gradients.

Comparison Table — Keras/TensorFlow vs PyTorch

9. Convolutional Neural Networks (CNNs)

What is it?

CNNs are a specialized neural network architecture designed specifically for image data, using "convolutional" layers that automatically detect visual patterns like edges, textures, and shapes.

Simple Conceptual Example (Keras)

python
from tensorflow import keras

cnn_model = keras.Sequential([
    keras.layers.Conv2D(32, (3, 3), activation="relu", input_shape=(64, 64, 3)),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(10, activation="softmax")
])

Explanation

• Conv2D layers scan across the image with small filters, learning to detect visual features like edges and textures.
• MaxPooling2D shrinks the data, keeping only the most important detected features, which reduces computation and helps the model generalize.
• Flatten() converts the resulting grid of features into a single long list, ready for regular Dense layers to make the final classification.

Real-World Example

CNNs power facial recognition, medical image analysis (detecting tumors in scans), and self-driving car vision systems.

10. Recurrent Neural Networks (RNNs) and LSTMs

What is it?

RNNs are designed for sequential data — where order matters, like sentences, time series, or audio. Unlike a regular neural network, an RNN has a form of "memory," using its own previous output as part of its next calculation.

The Problem RNNs Have — and LSTMs' Solution

Plain RNNs struggle to remember information from far back in a long sequence (a problem called "vanishing gradients"). LSTM (Long Short-Term Memory) networks are a specialized type of RNN designed specifically to remember important information over much longer sequences.

Simple Conceptual Example (Keras)

python
from tensorflow import keras

rnn_model = keras.Sequential([
    keras.layers.LSTM(64, input_shape=(10, 1)),   # sequence length 10, 1 feature per step
    keras.layers.Dense(1)
])

Real-World Example

RNNs/LSTMs are used for text generation, language translation, stock price prediction, and speech recognition — anywhere the order of the data carries meaning.

Important Points

• Use CNNs for image/spatial data; use RNNs/LSTMs for sequential/time-based data.
• Modern large language models (covered in the next file) largely use a newer architecture called "Transformers," which has mostly replaced RNNs/LSTMs for language tasks — but understanding RNNs/LSTMs is still valuable foundational knowledge.

11. Transfer Learning

What is it?

Transfer learning means taking a model already trained on a huge dataset (by someone else, often a large company with massive computing resources) and adapting it to your own, usually much smaller, specific task — instead of training a new model completely from scratch.

Definition: Transfer learning reuses a pre-trained model's learned features as a starting point for a new, related task, dramatically reducing the data and computation needed.

Real-World Example

A model pre-trained on millions of general images (like ImageNet) already understands general visual features (edges, shapes, textures). You can reuse those learned features and just retrain the final layers to recognize your own specific categories (e.g., identifying specific plant diseases from leaf photos), needing far less of your own data and training time.

Important Points

• Transfer learning is extremely common in practice — training a large model completely from scratch requires massive datasets and computing power most individuals and even many companies don't have.
• Especially popular for image classification (using pre-trained CNNs) and, as covered in the next file, for adapting large pre-trained language models.

Common Beginner Mistakes — Summary for This Section

• Forgetting an activation function entirely, which limits the network to only learning simple linear relationships regardless of how many layers it has.
• Choosing the wrong output activation function for the task (e.g., using Sigmoid for a multi-class problem instead of Softmax).
• Training for too few epochs (underfitting) or too many (overfitting) without monitoring performance on a validation set.
• Not normalizing/scaling input features, which can slow down or destabilize training.

Cheat Sheet — Deep Learning

python
# Keras/TensorFlow
from tensorflow import keras
model = keras.Sequential([
    keras.layers.Dense(8, activation="relu", input_shape=(n_features,)),
    keras.layers.Dense(1, activation="sigmoid")
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X, y, epochs=50)
model.predict(new_data)

# PyTorch
import torch.nn as nn
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(in_features, out_features)
    def forward(self, x):
        return self.layer(x)

loss.backward()      # backpropagation
optimizer.step()      # update weights

Interview Questions

Q1. What is the role of an activation function in a neural network? Answer: It introduces non-linearity, allowing the network to learn complex patterns — without it, stacking multiple layers would behave the same as a single linear equation.

Q2. What is backpropagation? Answer: The algorithm used to calculate how much each weight in the network contributed to the prediction error, working backward from the output layer, so the optimizer knows how to adjust each weight to reduce that error.

Q3. What is the difference between a CNN and an RNN? Answer: CNNs are designed for spatial data like images, using convolutional layers to detect visual patterns. RNNs (and LSTMs) are designed for sequential data like text or time series, where the order of the data matters.

Q4. What is transfer learning, and why is it useful? Answer: Reusing a model already pre-trained on a large dataset as a starting point for a new, related task — dramatically reducing the amount of data and computation needed compared to training entirely from scratch.

Q5. What is the difference between an epoch and a batch? Answer: An epoch is one complete pass through the entire training dataset. A batch is a smaller subset of the data processed together during one step of training within an epoch.

Q6. Why is Adam a popular choice of optimizer? Answer: It adapts its learning rate during training and generally converges faster and more reliably than basic gradient descent, with less manual tuning required.

Practice Questions

Beginner

1. Explain, in your own words, the difference between a neuron's weight and its bias.
2. List the three common activation functions covered and one situation where each would be used.
3. Build a simple Keras Sequential model with one hidden layer for a binary classification problem.
4. Explain why a loss function is necessary during training.
5. Explain the difference between supervised deep learning and the general concept of supervised learning from the Machine Learning file.

Intermediate

1. Train the Keras pass/fail example from this file with a different number of hidden neurons, and observe if predictions change.
2. Build a PyTorch model with two hidden layers instead of one, for the same pass/fail dataset.
3. Explain, in your own words, why CNNs use pooling layers.
4. Explain the vanishing gradient problem and how LSTMs address it.
5. Research and briefly summarize one real-world application each of a CNN and an RNN/LSTM.

Challenge

1. Build and train a small Keras neural network on a slightly larger synthetic dataset (e.g., 100 samples) for a binary classification task, and evaluate its accuracy.
2. Compare the same classification problem solved with (a) Logistic Regression (from the Machine Learning file) and (b) a small neural network, and discuss when the added complexity of a neural network is actually justified.
3. Research a pre-trained model (like MobileNet or ResNet) and explain, in your own words, how you would apply transfer learning to adapt it for a custom image classification task.