Data Analysis (NumPy, Pandas, Matplotlib, Seaborn)

Data analysis is one of the biggest reasons Python became so popular. This file introduces the four essential libraries every data analyst, data scientist, and ML engineer uses daily: NumPy (numerical arrays), Pandas (tabular data), and Matplotlib/Seaborn (visualization).

bash
pip install numpy pandas matplotlib seaborn

1. NumPy — Numerical Arrays

What is it?

NumPy (Numerical Python) provides a fast, memory-efficient array type — the foundation almost every other data science library in Python is built on top of.

Definition: NumPy is a Python library for fast numerical computing, built around a powerful multi-dimensional array object.

Why do we use it?

Regular Python lists are flexible but slow for large-scale number crunching. NumPy arrays are much faster and support powerful operations — like adding two entire arrays together element-by-element — with a single line of code, instead of writing manual loops.

Creating Arrays

python
import numpy as np

arr = np.array([1, 2, 3, 4, 5])
print(arr)                # [1 2 3 4 5]
print(type(arr))           # <class 'numpy.ndarray'>

matrix = np.array([[1, 2, 3], [4, 5, 6]])
print(matrix)
# [[1 2 3]
#  [4 5 6]]

zeros = np.zeros((2, 3))     # 2x3 array of zeros
ones = np.ones((3, 3))        # 3x3 array of ones
range_arr = np.arange(0, 10, 2)   # [0 2 4 6 8]

Indexing and Slicing

python
arr = np.array([10, 20, 30, 40, 50])

print(arr[0])       # 10
print(arr[1:4])      # [20 30 40]

matrix = np.array([[1, 2, 3], [4, 5, 6]])
print(matrix[0, 2])   # 3 (row 0, column 2)
print(matrix[:, 1])   # [2 5] (all rows, column 1)

Array Operations (Vectorization)

python
arr = np.array([1, 2, 3, 4])

print(arr + 10)     # [11 12 13 14] - added to EVERY element at once
print(arr * 2)      # [2 4 6 8]
print(arr ** 2)      # [1 4 9 16]

a = np.array([1, 2, 3])
b = np.array([10, 20, 30])
print(a + b)          # [11 22 33] - element-wise addition

Explanation: This is called vectorization — operations apply to every element automatically, without writing an explicit loop. This is both far more concise and far faster than looping through a plain Python list.

Useful NumPy Functions

python
data = np.array([23, 45, 12, 67, 34, 89, 21])

print(np.mean(data))     # average
print(np.median(data))   # middle value
print(np.std(data))       # standard deviation
print(np.min(data))        # smallest value
print(np.max(data))        # largest value
print(np.sort(data))       # sorted array

reshaped = np.arange(12).reshape(3, 4)   # reshape into 3 rows, 4 columns
print(reshaped)

Common Mistakes

• Trying to add a Python list directly with +, expecting element-wise addition — regular lists concatenate with + instead; you need a NumPy array for element-wise math.
• Forgetting NumPy arrays must contain a single consistent data type (unlike Python lists, which can freely mix types).

Important Points

• NumPy arrays are the foundation of Pandas, and much of the scientific/ML Python ecosystem.
• Vectorized operations are both more concise and significantly faster than manual loops over large datasets.

Practice

1. Create a NumPy array of 10 numbers and print its mean, min, and max.
2. Create a 2D array (3x3) and print just its middle row.

2. Pandas — Working with Tabular Data

What is it?

Pandas provides the DataFrame — a table-like structure (rows and columns, similar to a spreadsheet or a SQL table) with powerful tools for cleaning, filtering, and analyzing data.

Definition: A DataFrame is a two-dimensional, labeled data structure in Pandas, similar to a spreadsheet or SQL table, used for organizing and analyzing tabular data.

Creating a DataFrame

python
import pandas as pd

data = {
    "name": ["Aditi", "Rohan", "Zara", "Karan"],
    "age": [21, 22, 20, 23],
    "marks": [85, 92, 78, 88]
}

df = pd.DataFrame(data)
print(df)

Output:

    name  age  marks
0  Aditi   21     85
1  Rohan   22     92
2   Zara   20     78
3  Karan   23     88

Reading Data From a CSV File

python
df = pd.read_csv("students.csv")

Exploring a DataFrame

python
print(df.head())        # first 5 rows
print(df.tail(3))        # last 3 rows
print(df.info())          # column types and non-null counts
print(df.describe())      # statistical summary (mean, min, max, etc.) of numeric columns
print(df.shape)            # (rows, columns)
print(df.columns)          # list of column names

Selecting Data

python
print(df["name"])                 # a single column (returns a Series)
print(df[["name", "marks"]])       # multiple columns (returns a DataFrame)
print(df.iloc[0])                   # first row, by position
print(df.loc[0, "name"])            # specific value, by label

Filtering Rows

python
high_scorers = df[df["marks"] > 80]
print(high_scorers)

adults_only = df[(df["age"] >= 21) & (df["marks"] > 80)]
print(adults_only)

Explanation: df["marks"] > 80 produces a series of True/False values; passing that inside df[...] keeps only the rows where the condition is True. Multiple conditions are combined with & (and) or | (or) — note the parentheses around each condition are required.

Sorting

python
print(df.sort_values("marks"))                  # ascending
print(df.sort_values("marks", ascending=False))  # descending

Adding and Modifying Columns

python
df["grade"] = df["marks"].apply(lambda x: "A" if x >= 85 else "B")
print(df)

Explanation: .apply() runs a function on every value in a column — here, a lambda function assigns a letter grade based on the marks value.

Grouping Data

python
data2 = {
    "name": ["Aditi", "Rohan", "Zara", "Karan"],
    "course": ["CS", "AI", "CS", "AI"],
    "marks": [85, 92, 78, 88]
}
df2 = pd.DataFrame(data2)

print(df2.groupby("course")["marks"].mean())

Output:

course
AI    90.0
CS    81.5
Name: marks, dtype: float64

Explanation: groupby("course") groups rows sharing the same course, and ["marks"].mean() calculates the average marks within each group — directly parallel to SQL's GROUP BY.

Handling Missing Values

python
df3 = pd.DataFrame({"name": ["Aditi", "Rohan", None], "marks": [85, None, 78]})

print(df3.isnull())              # shows True/False for missing values
print(df3.isnull().sum())         # count of missing values per column

df3_dropped = df3.dropna()         # remove rows with ANY missing value
df3_filled = df3.fillna(0)          # replace missing values with 0

Explanation: Real-world data is almost never perfectly clean — .isnull() helps identify gaps, and .dropna()/.fillna() are the two most common strategies for handling them (either remove the incomplete rows, or fill in a reasonable default value).

Merging DataFrames

python
students = pd.DataFrame({"id": [1, 2, 3], "name": ["Aditi", "Rohan", "Zara"]})
marks = pd.DataFrame({"id": [1, 2, 3], "marks": [85, 92, 78]})

merged = pd.merge(students, marks, on="id")
print(merged)

Output:

   id   name  marks
0   1  Aditi     85
1   2  Rohan     92
2   3   Zara     78

Explanation: pd.merge() combines two DataFrames based on a shared column — directly parallel to a SQL JOIN.

Common Mistakes

• Forgetting .copy() when creating a filtered/modified DataFrame from another, sometimes triggering a confusing SettingWithCopyWarning.
• Confusing .loc[] (label-based access) with .iloc[] (position-based access).
• Forgetting parentheses around each condition when combining filters with &/|.

Important Points

• Pandas is the standard tool for cleaning, exploring, and analyzing tabular data in Python.
• .groupby(), .merge(), .dropna()/.fillna() cover the vast majority of everyday data-cleaning tasks.

Practice

1. Load a CSV of your choice (or create a small DataFrame manually) and print its summary statistics using .describe().
2. Filter a DataFrame of students to show only those with marks above 75, sorted in descending order.

3. Matplotlib — Basic Visualization

What is it?

Matplotlib is Python's foundational plotting library — used to create line charts, bar charts, scatter plots, histograms, and more.

Simple Examples

python
import matplotlib.pyplot as plt

# Line chart
months = ["Jan", "Feb", "Mar", "Apr"]
sales = [200, 250, 180, 300]

plt.plot(months, sales, marker="o")
plt.title("Monthly Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.show()

# Bar chart
students = ["Aditi", "Rohan", "Zara"]
marks = [85, 92, 78]

plt.bar(students, marks, color="skyblue")
plt.title("Student Marks")
plt.show()

# Scatter plot
plt.scatter([1, 2, 3, 4], [10, 20, 15, 25])
plt.title("Sample Scatter Plot")
plt.show()

# Histogram
data = [23, 45, 12, 67, 34, 89, 21, 45, 33, 56]
plt.hist(data, bins=5)
plt.title("Distribution of Values")
plt.show()

Explanation

• plt.plot(), plt.bar(), plt.scatter(), and plt.hist() each create a different chart type suited to different data: trends over time (line), comparing categories (bar), relationships between two variables (scatter), and distribution/frequency (histogram).
• plt.show() displays the chart — required to actually see the plot when running a .py script.

Important Points

• Choose the chart type based on what story you're telling: trends → line, comparisons → bar, relationships → scatter, distribution → histogram.

4. Seaborn — Statistical Visualization

What is it?

Seaborn is built on top of Matplotlib, offering more attractive default styling and simpler syntax specifically for statistical charts.

Simple Examples

python
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({
    "course": ["CS", "AI", "CS", "AI", "CS"],
    "marks": [85, 92, 78, 88, 95]
})

sns.barplot(x="course", y="marks", data=df)
plt.title("Average Marks by Course")
plt.show()

sns.histplot(df["marks"], bins=5, kde=True)
plt.title("Marks Distribution")
plt.show()

sns.boxplot(x="course", y="marks", data=df)
plt.title("Marks Spread by Course")
plt.show()

Explanation

• sns.barplot() automatically calculates and displays averages grouped by category — no manual groupby() needed first.
• kde=True in histplot() overlays a smooth density curve on top of the histogram, showing the overall distribution shape more clearly.
• sns.boxplot() shows the spread, median, and potential outliers of a numeric variable, broken down by category.

Comparison Table — Matplotlib vs Seaborn

Common Mistakes

• Forgetting plt.show() when running plotting code as a .py script (not needed in some interactive environments like Jupyter, but required in plain scripts).
• Not labeling axes/titles, making charts hard to interpret for anyone besides the person who made them.

Common Beginner Mistakes — Summary for This Section

• Confusing NumPy arrays' element-wise math with Python list concatenation.
• Confusing .loc[] with .iloc[] in Pandas.
• Forgetting to handle missing values before analyzing data.
• Forgetting plt.show() in script-based Matplotlib/Seaborn code.

Cheat Sheet — Data Analysis

python
import numpy as np
arr = np.array([1, 2, 3])
np.mean(arr); np.sum(arr); np.sort(arr)

import pandas as pd
df = pd.read_csv("file.csv")
df.head(); df.info(); df.describe()
df[df["col"] > 10]                        # filter
df.sort_values("col", ascending=False)     # sort
df.groupby("col")["other_col"].mean()      # group
df.dropna(); df.fillna(0)                   # missing values
pd.merge(df1, df2, on="key")                # join

import matplotlib.pyplot as plt
plt.plot(x, y); plt.bar(x, y); plt.hist(data)
plt.title(""); plt.xlabel(""); plt.ylabel(""); plt.show()

import seaborn as sns
sns.barplot(x="col1", y="col2", data=df)
sns.histplot(df["col"], kde=True)
sns.boxplot(x="col1", y="col2", data=df)

Mini Project: Student Performance Analysis

Objective

Analyze a small dataset of student marks — calculating statistics, identifying top and struggling students, and visualizing the results.

Requirements

• Load student data (name, subject scores) into a DataFrame.
• Calculate each student's average and identify pass/fail.
• Visualize the class's overall performance.

Concepts Used

Pandas (DataFrames, filtering, grouping), Matplotlib/Seaborn visualization.

Complete Code

python
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = {
    "name": ["Aditi", "Rohan", "Zara", "Karan", "Meera"],
    "maths": [85, 45, 92, 38, 76],
    "science": [78, 55, 88, 42, 81],
    "english": [90, 60, 85, 50, 70]
}

df = pd.DataFrame(data)

# Calculate average marks per student
df["average"] = df[["maths", "science", "english"]].mean(axis=1)

# Determine pass/fail (pass mark: 40 average)
df["result"] = df["average"].apply(lambda avg: "Pass" if avg >= 40 else "Fail")

print(df)

topper = df.loc[df["average"].idxmax()]
print(f"\nTopper: {topper['name']} with an average of {topper['average']:.2f}")

failing_students = df[df["result"] == "Fail"]
print(f"\nStudents who failed:\n{failing_students[['name', 'average']]}")

# Visualization
sns.barplot(x="name", y="average", data=df, palette="viridis")
plt.title("Average Marks per Student")
plt.axhline(y=40, color="red", linestyle="--", label="Pass Mark")
plt.legend()
plt.show()

Code Explanation

• df[["maths", "science", "english"]].mean(axis=1) calculates the average across columns for each row (student) — axis=1 means "average across this row," not down a column.
• .idxmax() finds the index (row position) of the highest average, letting us look up the topper's full row.
• The bar chart visualizes every student's average, with a horizontal reference line marking the pass threshold.

Sample Output

    name  maths  science  english    average result
0  Aditi     85       78       90  84.333333   Pass
1  Rohan     45       55       60  53.333333   Pass
2   Zara     92       88       85  88.333333   Pass
3  Karan     38       42       50  43.333333   Pass
4  Meera     76       81       70  75.666667   Pass

Topper: Zara with an average of 88.33

Possible Improvements

• Add subject-wise topper identification, not just overall.
• Load data from a real CSV file instead of a hardcoded dictionary.
• Add a correlation analysis between subjects (do students who do well in Maths also do well in Science?).

Challenge Task

Extend the analysis to calculate and visualize the class average per subject (not just per student), identifying which subject the class struggled with most.

Interview Questions

Q1. What is the main advantage of NumPy arrays over Python lists? Answer: NumPy arrays support fast, vectorized mathematical operations across entire arrays at once, and are more memory-efficient — both significant advantages for large-scale numerical computation compared to looping through plain Python lists.

Q2. What is a Pandas DataFrame? Answer: A two-dimensional, labeled data structure (rows and columns) similar to a spreadsheet or SQL table, used for organizing, cleaning, and analyzing tabular data.

Q3. What is the difference between `.loc[]` and `.iloc[]` in Pandas? Answer: .loc[] accesses data by label (e.g., column name or index label). .iloc[] accesses data by integer position, regardless of labels.

Q4. How do you handle missing values in a Pandas DataFrame? Answer: Using .isnull() to detect them, then either .dropna() to remove rows/columns with missing values, or .fillna(value) to replace them with a specified default.

Q5. What is the difference between Matplotlib and Seaborn? Answer: Matplotlib is the foundational, lower-level plotting library giving full manual control. Seaborn is built on top of it, offering more attractive default styles and simpler syntax specifically for statistical visualizations, and integrates very naturally with Pandas DataFrames.

Q6. What does `groupby()` do in Pandas? Answer: It groups rows sharing a common value in a specified column, typically combined with an aggregate function (like .mean() or .sum()) to summarize data within each group — directly analogous to SQL's GROUP BY.

Practice Questions

Beginner

1. Create a NumPy array of 10 random-looking numbers and calculate their mean and standard deviation.
2. Create a Pandas DataFrame of 5 products with name and price, and print .describe().
3. Filter a DataFrame of employees to show only those earning above a certain salary.
4. Create a simple bar chart of 5 categories and their values using Matplotlib.
5. Create a histogram of 20 random-looking numbers using Seaborn.

Intermediate

1. Load a CSV file of your choice into a DataFrame and print the top 5 rows, column info, and summary statistics.
2. Group a DataFrame of sales data by "region" and calculate total sales per region.
3. Handle missing values in a DataFrame by filling numeric columns with their mean value.
4. Merge two DataFrames (e.g., orders and customers) on a shared customer_id column.
5. Create a Seaborn box plot comparing a numeric column across different categories.

Challenge

1. Analyze a dataset of exam scores across multiple subjects: find the topper, the subject with the lowest class average, and visualize all subject averages in one chart.
2. Build a small sales dashboard: load sales data, calculate monthly totals, and visualize the trend using a line chart.
3. Extend the Student Performance Analysis mini project to categorize each student into grade bands (A/B/C/D/Fail) and visualize how many students fall into each band using a bar chart.