Pandas

1. Introduction

Real-world data almost never comes as a clean list of numbers — it comes as spreadsheets, CSV files, and tables with labeled columns and rows. Pandas is the library that makes working with this kind of structured, table-like data easy in Python. If NumPy is about fast numbers, Pandas is about organizing and exploring real datasets — which is exactly what you'll do at the start of every ML project.

2. What is Pandas?

Simple definition: Pandas is a Python library used to load, explore, clean, and manipulate table-like data (rows and columns), similar to an Excel spreadsheet.

Technical explanation: Pandas provides two core data structures: the Series (a single labeled column of data) and the DataFrame (a two-dimensional labeled table made up of rows and columns), along with powerful tools for reading, filtering, transforming, and summarizing data.

3. Why is it Important?

• Nearly every ML project begins by loading a dataset into a Pandas DataFrame.
• Pandas makes data cleaning, filtering, and transformation dramatically easier than using plain Python or NumPy alone.
• It integrates directly with Scikit-learn, Matplotlib, and Seaborn for the entire data science workflow.

4. Prerequisites

You should be comfortable with Python basics and NumPy (Topic 5), since Pandas is built on top of NumPy internally.

5. Core Concepts

1. Series — a single labeled column of data
2. DataFrame — a labeled table of rows and columns
3. Reading data (e.g., from CSV files)
4. Viewing and exploring data (head(), info(), describe())
5. Selecting rows and columns
6. Filtering data with conditions
7. Handling missing values (introductory view — covered in depth in Module 3)
8. Adding and modifying columns

6. Detailed Explanation

a) Series

A Series is essentially a single column of data with an index (labels) attached to each value — think of it as one labeled column from a spreadsheet.

b) DataFrame

A DataFrame is a full table made of multiple Series (columns) sharing the same row index — this is the primary structure you'll use throughout your ML journey.

c) Reading Data

pd.read_csv("filename.csv") loads data from a CSV file directly into a DataFrame, ready for exploration.

d) Viewing and Exploring Data

• df.head() shows the first few rows.
• df.info() shows column names, data types, and non-null counts.
• df.describe() shows summary statistics (mean, min, max, etc.) for numeric columns.

e) Selecting Rows and Columns

You can select a single column using df["column_name"], multiple columns using a list of names, and specific rows using .loc[] (label-based) or .iloc[] (position-based).

f) Filtering Data

You can filter rows based on a condition, e.g., df[df["age"] > 18] selects only rows where the age column is greater than 18.

g) Handling Missing Values (Introductory)

df.isnull() identifies missing values, and df.dropna() or df.fillna() can remove or fill them respectively. (This is explored in full detail in Module 3 — Data Preprocessing.)

h) Adding and Modifying Columns

You can create a new column simply by assigning a value or an expression to a new column name, e.g., df["new_column"] = df["old_column"] * 2.

7. How It Works

1. Pandas reads structured data (like a CSV file) and organizes it into rows and columns inside a DataFrame.
2. Each column is internally handled like a NumPy array, so operations on columns are fast.
3. Pandas keeps track of row and column labels, letting you reference data by name instead of only by position.
4. Operations like filtering or adding columns return new, updated views of the data based on your instructions.

8. Real-World Example

Think of a DataFrame like an Excel spreadsheet of student records — one row per student, with columns for name, age, and marks. Just like you'd use Excel filters to see only students who scored above 80, Pandas lets you do the exact same thing with a single line of code.

9. Technical Example

python
import pandas as pd

data = {"name": ["Aarav", "Meera"], "marks": [78, 92]}
df = pd.DataFrame(data)
print(df)

Here, a Python dictionary is converted into a proper table (DataFrame), with "name" and "marks" as columns.

10. Python Example

python
import pandas as pd

# Creating a DataFrame from a dictionary
data = {
    "name": ["Aarav", "Meera", "Kabir", "Diya"],
    "age": [21, 20, 22, 19],
    "marks": [78, 92, 65, 88]
}
df = pd.DataFrame(data)

print("Full DataFrame:")
print(df)

# Viewing the first 2 rows
print("\nFirst 2 rows:")
print(df.head(2))

# Getting summary statistics
print("\nSummary statistics:")
print(df.describe())

# Selecting a single column
print("\nMarks column:")
print(df["marks"])

# Filtering rows where marks > 75
print("\nStudents scoring above 75:")
print(df[df["marks"] > 75])

# Adding a new column
df["passed"] = df["marks"] >= 40
print("\nDataFrame with 'passed' column:")
print(df)

# Selecting rows using .loc[]
print("\nRow at index 1 using loc:")
print(df.loc[1])

Expected Output:

text
Full DataFrame:
    name  age  marks
0  Aarav   21     78
1  Meera   20     92
2  Kabir   22     65
3   Diya   19     88

First 2 rows:
    name  age  marks
0  Aarav   21     78
1  Meera   20     92

Summary statistics:
             age      marks
count   4.000000   4.000000
mean   20.500000  80.750000
...

Marks column:
0    78
1    92
2    65
3    88
Name: marks, dtype: int64

Students scoring above 75:
    name  age  marks
0  Aarav   21     78
1  Meera   20     92
3   Diya   19     88

DataFrame with 'passed' column:
    name  age  marks  passed
0  Aarav   21     78    True
1  Meera   20     92    True
2  Kabir   22     65    True
3   Diya   19     88    True

Row at index 1 using loc:
name     Meera
age         20
marks       92
Name: 1, dtype: object

11. Code Explanation

• pd.DataFrame(data) converts the dictionary into a proper labeled table, where dictionary keys become column names.
• df.head(2) displays only the first two rows — useful for quickly previewing large datasets.
• df.describe() automatically calculates useful statistics (count, mean, standard deviation, min, max, etc.) for numeric columns.
• df["marks"] selects just the marks column as a Series.
• df[df["marks"] > 75] filters and returns only the rows where the condition is True — this pattern is used constantly in real ML data cleaning.
• df["passed"] = df["marks"] >= 40 creates a brand-new column based on a condition applied to an existing column.
• df.loc[1] retrieves the entire row labeled with index 1, showing all column values for that specific student.

12. Advantages

• Makes loading, exploring, and cleaning real-world datasets fast and intuitive.
• Provides powerful filtering and selection tools with minimal code.
• Integrates seamlessly with NumPy, Matplotlib, Seaborn, and Scikit-learn.
• Handles missing data and different data types gracefully.

13. Limitations

• Can consume significant memory with very large datasets (millions of rows).
• Some operations can be slower than pure NumPy for extremely large numeric-only data.
• The many available methods can feel overwhelming to complete beginners at first.

14. Common Mistakes

• Forgetting that df["marks"] > 75 returns True/False values, and that you need df[condition] to actually filter rows.
• Confusing .loc[] (label-based selection) with .iloc[] (position-based selection).
• Modifying a DataFrame and expecting the original variable to update automatically without reassignment (depending on the operation).
• Not checking df.info() early, and missing important issues like wrong data types or missing values.

15. Best Practices

• Always inspect a new dataset first using df.head(), df.info(), and df.describe().
• Use clear, descriptive column names.
• Use .loc[] for label-based access and .iloc[] for position-based access, to avoid confusion.
• Check for missing values early, before moving into deeper analysis or modeling.

16. Real-World Applications

• Loading and exploring datasets such as sales records, survey results, or sensor readings before building ML models.
• Cleaning messy real-world data (missing values, incorrect formats) before analysis.
• Preparing features and labels for machine learning pipelines.

17. Interview-Oriented Points

• Be ready to explain the difference between a Series and a DataFrame.
• Understand the difference between .loc[] and .iloc[].
• Know how to filter rows based on a condition.
• Be able to explain what df.describe() and df.info() are used for.

18. Exam-Oriented Points

• Series = single labeled column; DataFrame = full labeled table.
• pd.read_csv() loads data from a CSV file into a DataFrame.
• .loc[] selects by label; .iloc[] selects by integer position.
• df.describe() gives summary statistics; df.info() gives structure and data type details.

19. Comparison Table — NumPy vs Pandas

20. Quick Revision

• Pandas provides Series (single column) and DataFrame (full table) structures.
• pd.read_csv() loads data from files; head(), info(), and describe() help explore it.
• Columns are selected with df["column"]; rows are filtered using conditions like df[df["col"] > value].
• .loc[] selects by label; .iloc[] selects by position.
• New columns can be created directly by assigning values or expressions to a new column name.