Multiple Linear Regression

1. Introduction

Real-world predictions rarely depend on just one factor — a house's price depends on its size, location, age, and more, all at once. Multiple Linear Regression extends the simple, single-feature version from Topic 1 to handle several input features simultaneously.

2. What is Multiple Linear Regression?

Simple definition: Multiple Linear Regression predicts a numeric output using two or more input features, combining them in a weighted sum to make a prediction.

Technical explanation: Multiple Linear Regression models the relationship between several independent variables (x₁, x₂, ..., xₙ) and a dependent variable y using the equation y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ, where the algorithm learns the optimal coefficients (b₀, b₁, ..., bₙ) that minimize the overall Mean Squared Error across the training data.

3. Why is it Important?

• Most real-world prediction problems naturally involve multiple relevant features, not just one.
• It remains simple and interpretable, while significantly increasing predictive power over Simple Linear Regression.
• It's a common baseline model used across countless business and scientific applications.

4. Prerequisites

Comfort with Simple Linear Regression (Topic 1) and Linear Algebra basics (Module 1, Topic 10).

5. Core Concepts

1. The multiple regression equation
2. Coefficients for each feature
3. Interpreting coefficients
4. Multicollinearity (a caution, not a deep dive)

6. Detailed Explanation

a) The Multiple Regression Equation

Instead of a single slope, Multiple Linear Regression learns one coefficient per input feature, combining them: y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ, where b₀ is the intercept and each bᵢ represents that feature's individual contribution.

b) Coefficients for Each Feature

Each coefficient (b₁, b₂, etc.) represents how much the output changes for a one-unit increase in that specific feature, holding all other features constant.

c) Interpreting Coefficients

A larger coefficient (in absolute value) suggests that feature has a stronger relationship with the output — though this depends on the scale of the feature, so features should generally be scaled (Module 3, Topic 7) before comparing coefficient sizes directly.

d) Multicollinearity

When two or more input features are highly correlated with each other, it becomes difficult to isolate each feature's individual effect on the output — this is called multicollinearity, and it can make coefficients unstable or hard to interpret (Feature Selection, Module 3 Topic 8, helps address this).

7. How It Works

1. Collect training data with multiple input features (x₁, x₂, ..., xₙ) and a known numeric output (y).
2. The algorithm finds the coefficients (b₀, b₁, ..., bₙ) that minimize Mean Squared Error across all training examples.
3. The resulting equation is the trained model.
4. For a new example, plug all its feature values into the equation to get a predicted output.

8. Real-World Example

Predicting a used car's price might depend on multiple factors simultaneously: its age, mileage, and engine size. Multiple Linear Regression can combine all three into a single equation — e.g., price = 20000 − 800×age − 0.05×mileage + 500×engine_size — capturing how each factor independently pulls the predicted price up or down.

9. Mathematical Explanation

Multiple Regression Equation:

y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ

Where:

• y = predicted output
• x₁, x₂, ..., xₙ = input features
• b₀ = intercept
• b₁, b₂, ..., bₙ = coefficients for each respective feature

Numerical Example:

Suppose a trained model learns: price = 50 + 10×size + 5×rooms

For a house with size = 20 and rooms = 3:

price = 50 + (10 × 20) + (5 × 3) = 50 + 200 + 15 = 265

Interpreting the Result: Holding rooms constant, each additional unit of size adds 10 to the price. Holding size constant, each additional room adds 5 to the price. The base price (when size=0 and rooms=0) would be 50, though this may not represent a realistic scenario in practice.

10. Python Example

python
import numpy as np
from sklearn.linear_model import LinearRegression

# Features: [size, num_rooms]
X = np.array([
    [10, 2], [15, 3], [20, 3], [25, 4], [30, 4]
])
y = np.array([200, 250, 280, 340, 360])

model = LinearRegression()
model.fit(X, y)

print("Intercept (b0):", model.intercept_)
print("Coefficients [size, rooms]:", model.coef_)

new_house = np.array([[22, 3]])
predicted_price = model.predict(new_house)
print("Predicted price:", predicted_price[0])

Expected Output (approximate — exact values depend on the least-squares fit):

text
Intercept (b0): 45.71
Coefficients [size, rooms]: [8.86 8.00]
Predicted price: 269.85

11. Code Explanation

• X now contains two columns — size and num_rooms — representing two input features per training example.
• model.fit(X, y) learns one coefficient for each feature, plus a single intercept.
• model.coef_ returns an array with one coefficient per feature, in the same order as the columns in X.
• model.predict(new_house) combines both feature values with their respective learned coefficients (plus the intercept) to generate a single predicted price.

12. Advantages

• Captures the combined effect of multiple relevant factors, improving predictive accuracy over single-feature models.
• Remains simple, fast, and interpretable compared to more complex algorithms.
• Coefficients provide clear insight into each feature's individual contribution.

13. Limitations

• Still assumes a fundamentally linear relationship between features and the output.
• Multicollinearity among features can make coefficients unstable or misleading.
• Sensitive to outliers, just like Simple Linear Regression.
• Adding many irrelevant features can hurt performance and interpretability (Feature Selection, Module 3, helps address this).

14. Common Mistakes

• Interpreting a large coefficient as "most important" without first scaling features to a comparable range.
• Ignoring multicollinearity, which can make individual coefficient interpretations unreliable.
• Including irrelevant or redundant features without applying feature selection first.
• Assuming Multiple Linear Regression can capture non-linear relationships between features and the output.

15. Best Practices

• Scale features (Module 3, Topic 7) before comparing coefficient magnitudes directly.
• Check for multicollinearity (e.g., via a correlation matrix, Module 3 Topic 8) before trusting individual coefficient interpretations.
• Apply feature selection to remove irrelevant or highly redundant features first.
• Validate the model using proper train-test splitting and evaluation metrics (Module 6).

16. Real-World Applications

• Predicting house prices using size, location, age, and number of rooms together.
• Forecasting sales based on advertising spend across multiple channels (TV, online, print).
• Estimating a person's expected salary based on experience, education level, and industry.

17. Interview-Oriented Points

• Be ready to explain how Multiple Linear Regression extends Simple Linear Regression.
• Understand what each coefficient represents (holding other features constant).
• Be able to explain multicollinearity and why it's a concern.

18. Exam-Oriented Points

• Multiple Linear Regression: y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ.
• Each coefficient represents that feature's individual effect, holding other features constant.
• Multicollinearity (highly correlated features) can make coefficients unstable and harder to interpret.

19. Comparison Table — Simple vs Multiple Linear Regression

20. Quick Revision

• Multiple Linear Regression predicts an output using two or more input features combined in a weighted sum.
• Each coefficient represents that feature's individual effect, holding other features constant.
• Multicollinearity (highly correlated features) can make coefficient interpretation unreliable.
• Feature scaling and feature selection are important best practices before and after fitting this model.