Linear Algebra Basics

1. Introduction

This is the final topic of Module 1, and an important one: linear algebra is the mathematical language behind how ML models actually store and process data internally. Every dataset, every set of model weights, and every neural network layer is represented using vectors and matrices. You don't need to become a mathematician — but understanding these basic building blocks will make everything from Linear Regression to Deep Learning far easier to understand later.

2. What is Linear Algebra?

Simple definition: Linear algebra is the branch of mathematics dealing with vectors (lists of numbers) and matrices (grids of numbers), and the rules for combining and transforming them.

Technical explanation: Linear algebra studies vector spaces and linear transformations, using structured numerical objects — scalars, vectors, and matrices — along with defined operations like addition, scalar multiplication, dot products, and matrix multiplication.

3. Why is it Important?

• Datasets are represented internally as matrices (rows = samples, columns = features).
• Model parameters (weights) in algorithms like Linear Regression and Neural Networks are stored and updated as vectors/matrices.
• Operations like the dot product are used constantly in ML — for example, to calculate a weighted sum of inputs.

4. Prerequisites

Basic arithmetic and comfort with NumPy (Topic 5) will make the Python examples in this topic much easier to follow.

5. Core Concepts

1. Scalars
2. Vectors
3. Matrices
4. Vector and matrix addition
5. Scalar multiplication
6. Dot product
7. Matrix multiplication
8. Transpose
9. Identity matrix

6. Detailed Explanation

a) Scalars

A scalar is simply a single number (e.g., 5, -2.3). Most everyday arithmetic operates on scalars.

b) Vectors

A vector is an ordered list of numbers, often representing a point in space or a set of feature values (e.g., [2, 4, 6]). Vectors have a "direction" and "magnitude" conceptually, though for ML purposes, you can often just think of them as ordered lists of numbers.

c) Matrices

A matrix is a grid of numbers arranged in rows and columns (e.g., a table of data where each row is a sample and each column is a feature).

d) Vector and Matrix Addition

Adding two vectors (or matrices) of the same shape simply adds their corresponding elements together.

e) Scalar Multiplication

Multiplying a vector or matrix by a scalar multiplies every individual element by that scalar.

f) Dot Product

The dot product of two vectors multiplies their corresponding elements together and then sums the results, producing a single number. This is used constantly in ML to calculate weighted sums (e.g., in Linear Regression and neural networks).

g) Matrix Multiplication

Matrix multiplication combines two matrices using a specific rule (each element of the result is a dot product of a row from the first matrix and a column from the second) — it is NOT the same as multiplying elements one-to-one.

h) Transpose

The transpose of a matrix flips its rows into columns (and vice versa). It's commonly used to reshape data for compatible matrix operations.

i) Identity Matrix

The identity matrix is a special square matrix with 1s along its diagonal and 0s everywhere else. Multiplying any matrix by the identity matrix leaves it unchanged — similar to how multiplying any number by 1 leaves it unchanged.

7. How It Works

For a dot product of two vectors [a, b, c] and [x, y, z]:

1. Multiply corresponding elements: a×x, b×y, c×z.
2. Sum these products together: (a×x) + (b×y) + (c×z).

For matrix multiplication of an (m×n) matrix with an (n×p) matrix:

1. The number of columns in the first matrix must match the number of rows in the second.
2. Each element in the resulting matrix is the dot product of a row from the first matrix and a column from the second.
3. The result is an (m×p) matrix.

8. Real-World Example

Think of a vector as a shopping cart's item quantities, e.g., [3, 2, 1] meaning 3 apples, 2 breads, 1 milk. If prices are [10, 20, 50] per item, the total bill is exactly the dot product: (3×10) + (2×20) + (1×50) = 30 + 40 + 50 = 120. This "multiply and sum" pattern is precisely how ML models combine input features with weights to make predictions.

9. Technical Example

python
import numpy as np

vector_a = np.array([2, 3])
vector_b = np.array([4, 5])

dot_product = np.dot(vector_a, vector_b)
print(dot_product)

This computes (2×4) + (3×5) = 8 + 15 = 23.

10. Mathematical Explanation

Dot Product Formula:

For vectors a = [a₁, a₂, ..., aₙ] and b = [b₁, b₂, ..., bₙ]:

a · b = (a₁×b₁) + (a₂×b₂) + ... + (aₙ×bₙ)

Where:

• a₁, a₂, ..., aₙ = elements of vector a
• b₁, b₂, ..., bₙ = elements of vector b
• n = number of elements in each vector (must be equal)

Numerical Example:

Let a = [2, 3, 4] and b = [1, 0, 5]

a · b = (2×1) + (3×0) + (4×5) = 2 + 0 + 20 = 22

Interpreting the Result: The dot product combines two vectors into a single number that reflects how their corresponding values interact — a concept used directly when a model multiplies input features by learned weights and sums the result.

11. Python Example

python
import numpy as np

# Scalars
scalar_a = 5
scalar_b = 3
print("Scalar addition:", scalar_a + scalar_b)

# Vectors
vector_a = np.array([2, 4, 6])
vector_b = np.array([1, 3, 5])

print("Vector addition:", vector_a + vector_b)
print("Scalar multiplication (vector_a * 2):", vector_a * 2)
print("Dot product:", np.dot(vector_a, vector_b))

# Matrices
matrix_a = np.array([[1, 2], [3, 4]])
matrix_b = np.array([[5, 6], [7, 8]])

print("Matrix A:\n", matrix_a)
print("Matrix B:\n", matrix_b)

print("Matrix addition:\n", matrix_a + matrix_b)
print("Matrix multiplication:\n", np.matmul(matrix_a, matrix_b))

# Transpose
print("Transpose of Matrix A:\n", matrix_a.T)

# Identity matrix
identity = np.eye(2)
print("Identity Matrix:\n", identity)
print("Matrix A multiplied by Identity:\n", np.matmul(matrix_a, identity))

Expected Output:

text
Scalar addition: 8
Vector addition: [3 7 11]
Scalar multiplication (vector_a * 2): [ 4  8 12]
Dot product: 44
Matrix A:
 [[1 2]
 [3 4]]
Matrix B:
 [[5 6]
 [7 8]]
Matrix addition:
 [[ 6  8]
 [10 12]]
Matrix multiplication:
 [[19 22]
 [43 50]]
Transpose of Matrix A:
 [[1 3]
 [2 4]]
Identity Matrix:
 [[1. 0.]
 [0. 1.]]
Matrix A multiplied by Identity:
 [[1. 2.]
 [3. 4.]]

12. Code Explanation

• vector_a + vector_b adds corresponding elements of two vectors together, producing a new vector of the same size.
• vector_a * 2 multiplies every element in the vector by 2 (scalar multiplication).
• np.dot(vector_a, vector_b) computes the dot product — multiplying corresponding elements and summing the results.
• np.matmul(matrix_a, matrix_b) performs true matrix multiplication, not simple element-by-element multiplication — this is a common point of confusion for beginners.
• matrix_a.T returns the transpose, flipping rows into columns.
• np.eye(2) creates a 2×2 identity matrix, and multiplying matrix_a by it returns matrix_a completely unchanged — just like multiplying a number by 1.

13. Advantages

• Provides an efficient, compact way to represent and process large datasets and model parameters.
• Enables extremely fast computation through optimized matrix operations (used internally by nearly all ML libraries).
• Forms the mathematical backbone that makes modern deep learning computationally feasible.

14. Limitations

• Concepts like matrix multiplication rules can feel abstract and confusing to beginners at first.
• Mismatched matrix/vector shapes are a very common source of errors in ML code.
• Visualizing operations beyond 2D or 3D vectors becomes difficult for human intuition.

15. Common Mistakes

• Confusing matrix multiplication (np.matmul() or @) with simple element-wise multiplication (*) — these produce very different results.
• Forgetting that matrix multiplication requires the number of columns in the first matrix to match the number of rows in the second.
• Assuming the dot product works on vectors of different lengths (it doesn't — they must match).
• Forgetting that transposing changes a matrix's shape, which can affect compatibility with other operations.

16. Best Practices

• Always check vector/matrix shapes before performing operations, especially multiplication.
• Use NumPy's built-in functions (np.dot(), np.matmul(), .T) instead of manually writing loops — they are faster and less error-prone.
• Visualize small examples by hand first to build intuition before trusting larger, more complex calculations.

17. Real-World Applications

• Representing datasets as matrices (rows = samples, columns = features) for ML models.
• Calculating weighted sums of inputs in Linear Regression and neural networks using dot products.
• Powering the massive matrix computations behind deep learning and image processing.

18. Interview-Oriented Points

• Be ready to explain the difference between element-wise multiplication and matrix multiplication.
• Understand what the dot product represents and where it's used in ML (e.g., weighted sums).
• Know the shape compatibility rule for matrix multiplication: (m×n) × (n×p) = (m×p).
• Be able to explain what the identity matrix does when used in multiplication.

19. Exam-Oriented Points

• A scalar is a single number; a vector is an ordered list; a matrix is a grid of numbers (rows × columns).
• Dot product: multiply corresponding elements, then sum the results.
• Matrix multiplication requires compatible shapes: columns of the first must equal rows of the second.
• The identity matrix leaves any compatible matrix unchanged when multiplied.

20. Comparison Table — Dot Product vs Element-Wise Multiplication

21. Quick Revision

• A scalar is a single number; a vector is a list of numbers; a matrix is a grid of numbers.
• Vector/matrix addition combines corresponding elements; scalar multiplication scales every element.
• Dot product = multiply corresponding elements, then sum → produces a single number.
• Matrix multiplication combines rows and columns using dot products, requiring compatible shapes.
• The identity matrix acts like the number 1 in multiplication — it leaves matrices unchanged.