Linear Regression

1. Introduction

Linear Regression is the simplest and most fundamental supervised learning algorithm, and often the very first real ML model students build. It predicts a continuous numeric value based on a single input feature, by fitting a straight line through the data. Understanding it deeply will make every other algorithm in this module easier to grasp.

2. What is Linear Regression?

Simple definition: Linear Regression is an algorithm that predicts a numeric value by fitting a straight line through the relationship between one input feature and the output.

Technical explanation: Simple Linear Regression models the relationship between a single independent variable (feature) x and a dependent variable (target) y using the equation y = mx + b, where the algorithm learns the optimal values of m (slope) and b (intercept) that minimize the overall prediction error across the training data.

3. Why is it Important?

• It's the foundational regression algorithm — nearly every other regression technique builds on its core ideas.
• It's simple, fast, and highly interpretable — you can directly see how much the output changes per unit change in the input.
• It's a common baseline model against which more complex models are compared.

4. Prerequisites

Comfort with Basic Statistics and Linear Algebra (Module 1), and Supervised Learning concepts (Module 2).

5. Core Concepts

1. The line equation: y = mx + b
2. Slope (m) and intercept (b)
3. The cost function (Mean Squared Error)
4. Finding the "line of best fit"
5. Making predictions with the trained line

6. Detailed Explanation

a) The Line Equation

Linear Regression assumes the relationship between input x and output y can be approximated by a straight line: y = mx + b, where m is the slope (how steeply y changes as x increases) and b is the intercept (the value of y when x is 0).

b) The Cost Function

To find the "best" line, we need a way to measure how wrong a given line is. The most common cost function for Linear Regression is Mean Squared Error (MSE) — the average of the squared differences between actual and predicted values. A smaller MSE means a better-fitting line.

c) Finding the Line of Best Fit

The algorithm searches for the values of m and b that minimize the MSE across all training examples. This can be solved directly using a mathematical formula (Ordinary Least Squares) or iteratively using an optimization technique called Gradient Descent.

d) Making Predictions

Once m and b are learned, predicting a new value is simple: plug the new x into y = mx + b.

7. How It Works

1. Collect training data with one input feature (x) and a known numeric output (y).
2. Calculate the values of m and b that minimize the Mean Squared Error across all training points.
3. The resulting line (y = mx + b) is the trained model.
4. For any new input value, plug it into the equation to get a predicted output.

8. Real-World Example

Imagine predicting a person's weight based on their height. Using historical height-weight data, Linear Regression finds the straight line that best represents this relationship, letting you estimate a reasonable weight for a new person based solely on their height.

9. Mathematical Explanation

Line Equation:

y = mx + b

Where:

• y = predicted output
• x = input feature
• m = slope (how much y changes per unit of x)
• b = intercept (value of y when x = 0)

Slope and Intercept Formulas (Ordinary Least Squares):

m = Σ((x − x̄)(y − ȳ)) / Σ((x − x̄)²)

b = ȳ − m × x̄

Where:

• x̄ = mean of all x values
• ȳ = mean of all y values

Mean Squared Error (Cost Function):

MSE = (1/n) × Σ(yᵢ − ŷᵢ)²

Where:

• n = number of data points
• yᵢ = actual value
• ŷᵢ = predicted value

Numerical Example:

Data: x = [1, 2, 3, 4], y = [3, 5, 7, 9]

• x̄ = 2.5, ȳ = 6
• m = Σ((x−2.5)(y−6)) / Σ((x−2.5)²) = [(-1.5×-3)+(-0.5×-1)+(0.5×1)+(1.5×3)] / [2.25+0.25+0.25+2.25] = (4.5+0.5+0.5+4.5)/5 = 10/5 = 2
• b = 6 − (2×2.5) = 6 − 5 = 1

So the learned line is: y = 2x + 1

Interpreting the Result: For every 1-unit increase in x, y increases by 2 units (the slope), and when x = 0, y equals 1 (the intercept). Checking: at x=1, y = 2(1)+1 = 3 ✓, matching our data exactly, since this example was constructed as a perfectly linear relationship.

10. Python Example

python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Training data: house size (in 100s sq ft) vs price (in $1000s)
X = np.array([[10], [15], [20], [25], [30]])
y = np.array([200, 250, 300, 350, 400])

model = LinearRegression()
model.fit(X, y)

print("Slope (m):", model.coef_[0])
print("Intercept (b):", model.intercept_)

predictions = model.predict(X)
print("Predictions on training data:", predictions)

new_size = np.array([[22]])
predicted_price = model.predict(new_size)
print("Predicted price for size 22:", predicted_price[0])

print("MSE:", mean_squared_error(y, predictions))
print("R² Score:", r2_score(y, predictions))

Expected Output (approximate):

text
Slope (m): 10.0
Intercept (b): 100.0
Predictions on training data: [200. 250. 300. 350. 400.]
Predicted price for size 22: 320.0
MSE: 0.0
R² Score: 1.0

11. Code Explanation

• model.fit(X, y) performs the actual training — internally calculating the slope and intercept that minimize MSE.
• model.coef_[0] and model.intercept_ reveal the learned slope and intercept — here, the model discovered price = 10 × size + 100.
• model.predict(new_size) uses the learned equation to estimate a price for a new, unseen house size.
• mean_squared_error() and r2_score() (explored fully in Module 6) quantify how well the model's predictions matched the actual training values — here, both metrics show a perfect fit since this example data is perfectly linear.

12. Advantages

• Simple, fast to train, and highly interpretable.
• Provides a clear mathematical relationship between input and output.
• Works very well when the true relationship between variables is genuinely linear.

13. Limitations

• Assumes a strictly linear relationship — performs poorly on data with curved or complex patterns.
• Highly sensitive to outliers, which can significantly distort the fitted line.
• Only handles one input feature at a time (Multiple Linear Regression, covered next, addresses this).

14. Common Mistakes

• Applying Linear Regression to clearly non-linear data without any transformation.
• Not checking for and addressing outliers before training, given the algorithm's sensitivity to them.
• Confusing the slope's meaning — it represents change in y per unit change in x, not the value of y itself.

15. Best Practices

• Visualize your data with a scatter plot first to check if a linear relationship genuinely appears reasonable.
• Check for and handle outliers before training, given Linear Regression's sensitivity to them.
• Use evaluation metrics like R² and RMSE (Module 6) to assess fit quality, not just visual inspection.

16. Real-World Applications

• Predicting house prices based on size.
• Estimating sales based on advertising spend.
• Forecasting simple trends like temperature changes over time.

17. Interview-Oriented Points

• Be ready to explain the equation y = mx + b and what each term represents.
• Understand how Mean Squared Error is used to find the "best" line.
• Be able to explain why Linear Regression is sensitive to outliers.

18. Exam-Oriented Points

• Linear Regression models y = mx + b, learning the slope (m) and intercept (b) from data.
• MSE (Mean Squared Error) is the standard cost function used to measure and minimize prediction error.
• Predictions are made by plugging new x values into the learned equation.

19. Comparison Table — Simple Linear Regression vs Multiple Linear Regression (Preview)

20. Quick Revision

• Linear Regression fits a straight line (y = mx + b) to model the relationship between one input and one output.
• The slope and intercept are learned by minimizing Mean Squared Error (MSE).
• Predictions are made by plugging new input values into the learned equation.
• Linear Regression is simple and interpretable but assumes a genuinely linear relationship and is sensitive to outliers.