PyTorch

1. Introduction

TensorFlow/Keras (Topic 3) isn't the only major Deep Learning framework — PyTorch, developed by Meta (Facebook), is the other dominant choice, especially popular in research settings and increasingly in industry too. This topic introduces PyTorch's approach, drawing direct comparisons to what you just learned.

2. What is PyTorch?

Term: PyTorch

Simple definition: PyTorch is an open-source Deep Learning framework known for its flexible, intuitive, "Python-like" style of building and training neural networks.

In simple words: If Keras is like following a recipe with clear, pre-set steps, PyTorch is more like cooking freely in your own kitchen — slightly more hands-on, but giving you finer control over every ingredient and step.

Technical explanation: PyTorch provides tensor computation with GPU acceleration (similar to TensorFlow) alongside "autograd" — an automatic differentiation system that tracks operations performed on tensors and automatically computes gradients needed for backpropagation — combined with a more explicit, code-first style for defining network architectures (typically via Python classes) rather than Keras's more declarative Sequential style.

3. Why is it Important?

• It's the dominant framework in AI research and academia, meaning most new, cutting-edge techniques are published with PyTorch code first.
• Its explicit, Python-native style helps you understand exactly what's happening at each step, which some learners find more transparent than Keras's higher-level abstractions.
• Being comfortable with both PyTorch and TensorFlow/Keras (Topic 3) means you can read and adapt Deep Learning code from virtually any source.

4. Prerequisites

Comfort with Neural Networks (Topic 2) and TensorFlow/Keras (Topic 3), since this topic draws direct comparisons.

5. Core Concepts

1. PyTorch tensors
2. autograd — automatic gradient computation
3. Defining a network as a Python class (nn.Module)
4. The manual-style training loop

6. Detailed Explanation

a) PyTorch Tensors

Just like TensorFlow's tensors (Topic 3), PyTorch tensors are multi-dimensional, GPU-capable arrays — conceptually and practically very similar to NumPy arrays, with easy conversion between the two.

b) `autograd` — Automatic Differentiation

Term: Autograd

Simple definition: PyTorch's system for automatically tracking every mathematical operation performed on a tensor, so it can later automatically calculate exactly how to adjust each weight during backpropagation (Topic 2) — without you manually deriving any calculus yourself.

In simple words: It's like PyTorch quietly taking notes on every single calculation you perform, so that later, when you ask "how should I adjust things to reduce my error?", it can instantly answer using those notes — without you needing to redo any of the underlying math.

c) Defining a Network as a Python Class

Unlike Keras's Sequential([...]) list-style definition, PyTorch typically defines a network as a Python class inheriting from nn.Module, with an __init__ method (Module 1, Topic 4's OOP concepts) defining the layers, and a forward method explicitly describing how data flows through them — giving more explicit, visible control over the forward propagation process (Topic 2).

d) The Manual-Style Training Loop

While Keras's .fit() handles the entire training loop internally, PyTorch traditionally expects you to write the loop yourself: forward pass → calculate loss → backward pass (loss.backward(), powered by autograd) → update weights (optimizer.step()) — repeated manually for each batch and epoch, giving finer visibility and control over each individual step.

7. How It Works

1. Define the network's architecture as a Python class (layers in __init__, data flow in forward).
2. Choose a loss function and an optimizer.
3. For each training epoch, and each batch of data: run a forward pass, calculate the loss, call loss.backward() to compute gradients via autograd, then optimizer.step() to update the weights.
4. Repeat until training is complete, then use the trained network to make predictions on new data.

8. Real-World Example

A researcher developing a brand-new type of neural network architecture (something Keras's pre-built layers might not directly support) would likely reach for PyTorch specifically because of its explicit, flexible style — being able to precisely define exactly how each custom piece of the network should behave, rather than working within a more constrained, pre-packaged structure.

9. Python Example

python
import torch
import torch.nn as nn
import torch.optim as optim

# Defining a simple network as a class
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(4, 8)
        self.layer2 = nn.Linear(8, 1)
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.sigmoid(self.layer2(x))
        return x

model = SimpleNet()
loss_function = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Simplified training loop (illustrative)
X_train = torch.rand(20, 4)     # 20 samples, 4 features
y_train = torch.randint(0, 2, (20, 1)).float()

for epoch in range(20):
    predictions = model(X_train)          # forward pass
    loss = loss_function(predictions, y_train)
    optimizer.zero_grad()
    loss.backward()                        # backward pass (autograd)
    optimizer.step()                       # update weights

print("Final training loss:", loss.item())

Expected Output (approximate — exact value varies due to random data):

text
Final training loss: 0.612

10. Code Explanation

• class SimpleNet(nn.Module): defines the network using standard Python OOP (Module 1, Topic 4) — __init__ sets up the layers, and forward explicitly describes how input data flows through them.
• loss.backward() triggers autograd — PyTorch automatically calculates exactly how much each weight contributed to the loss, without any manual calculus.
• optimizer.step() then actually updates the weights, using those automatically-computed gradients.
• optimizer.zero_grad() resets accumulated gradients before each new pass — an easy-to-forget but essential step, since PyTorch accumulates gradients by default across calls.
• Notice how much MORE explicit this training loop is compared to Keras's single .fit() call — this is the core practical difference between the two frameworks' philosophies.

11. Advantages

• Explicit, transparent, "Python-native" style appeals to researchers and those who want fine-grained control.
• Dominant in the AI research community, meaning most cutting-edge papers and code are PyTorch-first.
• autograd handles the complex calculus of backpropagation automatically, despite the more manual training loop structure.

12. Limitations

• The more manual, explicit style has a steeper learning curve for complete beginners compared to Keras's simplified .fit() approach.
• Writing your own training loop means more code (and more opportunities for small bugs like forgetting optimizer.zero_grad()).
• Same fundamental Deep Learning limitations as Topic 1 (data/compute requirements, interpretability challenges) apply here too.

13. Common Mistakes

• Forgetting to call optimizer.zero_grad() before each backward pass, causing gradients to incorrectly accumulate across iterations.
• Confusing PyTorch's more manual training loop with Keras's single .fit() call, and expecting one framework's syntax to work in the other.
• Not understanding that forward() defines the actual computation, while __init__ only defines the available layers/components.

14. Best Practices

• Start with TensorFlow/Keras (Topic 3) if you're brand new to Deep Learning, given its gentler learning curve; move to PyTorch once you want finer control or need to read research code.
• Always remember optimizer.zero_grad() before each loss.backward() call.
• Structure networks as clear, well-organized classes with descriptive layer names.

15. Real-World Applications

• The dominant framework in academic AI research, including most published cutting-edge Deep Learning papers.
• Widely used in production at major tech companies for both research and deployed models.
• A common choice for building custom architectures for Computer Vision (Topic 5) and NLP (Topic 6) research.

16. Interview-Oriented Points

• Be ready to explain the key philosophical difference between PyTorch (explicit, manual training loop) and Keras (higher-level, .fit()-based).
• Understand what autograd does and why it eliminates the need for manual calculus.
• Be able to explain why optimizer.zero_grad() is necessary before each backward pass.

17. Exam-Oriented Points

• PyTorch networks are typically defined as Python classes inheriting from nn.Module, with __init__ (layers) and forward (data flow) methods.
• autograd automatically computes gradients needed for backpropagation.
• The manual training loop: forward pass → calculate loss → zero_grad() → backward() → optimizer.step().

18. Comparison Table — TensorFlow/Keras vs PyTorch

19. Quick Revision

• PyTorch is a flexible, Python-native Deep Learning framework, popular especially in research.
• autograd automatically computes gradients for backpropagation, tracking operations on tensors.
• Networks are typically defined as classes (nn.Module), with __init__ for layers and forward for data flow.
• Training requires a manual loop: forward pass → loss → zero_grad() → backward() → optimizer.step().