Loading Datasets

1. Introduction

Every ML project begins the same way: getting your data into Python so you can actually work with it. Real-world data lives in many different file formats — CSV, Excel, JSON, and databases — and this topic covers how to load each of them cleanly using Pandas, setting the stage for the entire preprocessing pipeline covered in this module.

2. What is Meant by "Loading Datasets"?

Simple definition: Loading a dataset means reading data from a file (or other source) into a Pandas DataFrame so you can inspect, clean, and use it in Python.

Technical explanation: Pandas provides a family of read_*() functions (read_csv, read_excel, read_json, read_sql, etc.) that parse structured data from various file formats and sources, converting them into a DataFrame ready for analysis.

3. Why is it Important?

• No preprocessing, analysis, or modeling can happen until data is successfully loaded into your program.
• Real-world datasets come in many formats, and each has small quirks (delimiters, encodings, sheet names) that can trip up beginners.
• A correctly loaded dataset with the right data types saves significant debugging time later in the pipeline.

4. Prerequisites

Comfort with Pandas basics (Module 1, Topic 6).

5. Core Concepts

1. Reading CSV files (pd.read_csv)
2. Reading Excel files (pd.read_excel)
3. Reading JSON files (pd.read_json)
4. Inspecting a newly loaded dataset
5. Common loading issues (delimiters, encodings, headers)

6. Detailed Explanation

a) Reading CSV Files

CSV (Comma-Separated Values) is the most common data format in ML. pd.read_csv("file.csv") loads it directly into a DataFrame.

b) Reading Excel Files

Excel files (.xlsx) are read using pd.read_excel("file.xlsx"), which requires the openpyxl library to be installed. You can also specify which sheet to load using the sheet_name parameter.

c) Reading JSON Files

JSON (JavaScript Object Notation) files, common in web data and APIs, are read using pd.read_json("file.json").

d) Inspecting a Newly Loaded Dataset

After loading, it's standard practice to immediately check df.shape, df.head(), df.info(), and df.columns to understand the dataset's structure before doing anything else.

e) Common Loading Issues

• Wrong delimiter (e.g., semicolon-separated files need pd.read_csv("file.csv", sep=";")).
• Encoding errors (sometimes requiring encoding="utf-8" or encoding="latin1").
• Missing or misaligned headers (fixed using the header parameter).

7. How It Works

1. Identify the file format and location of your dataset.
2. Choose the appropriate pd.read_*() function.
3. Load the data into a DataFrame variable.
4. Immediately inspect the result (shape, head(), info()) to confirm it loaded correctly.
5. Proceed to cleaning and preprocessing (covered in the following topics).

8. Real-World Example

Imagine receiving a customer dataset as a CSV export from a company's sales system. Before you can analyze sales trends or build a prediction model, you first need to load that CSV file into a DataFrame — this simple but essential first step is what this topic covers.

9. Python Example

python
import pandas as pd

# Reading a CSV file
df_csv = pd.read_csv("students.csv")

# Reading an Excel file (a specific sheet)
# df_excel = pd.read_excel("students.xlsx", sheet_name="Sheet1")

# Reading a JSON file
# df_json = pd.read_json("students.json")

# Inspecting the loaded dataset
print("Shape:", df_csv.shape)
print("\nFirst 5 rows:")
print(df_csv.head())
print("\nColumn info:")
print(df_csv.info())

Expected Output (structure will vary based on actual file content):

text
Shape: (150, 4)

First 5 rows:
    name  age  marks  passed
0  Aarav   21     78    True
1  Meera   20     92    True
...

Column info:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150 entries, 0 to 149
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   name     150 non-null    object
 1   age      150 non-null    int64
 2   marks    148 non-null    float64
 3   passed   150 non-null    bool

10. Code Explanation

• pd.read_csv("students.csv") reads the CSV file directly into a DataFrame — Pandas automatically detects column names from the first row (header) by default.
• df.shape immediately tells you the number of rows and columns, giving a quick sense of dataset size.
• df.head() previews the first 5 rows, helping you visually confirm the data loaded as expected.
• df.info() reveals data types and non-null counts per column — notice how marks shows 148 non-null out of 150 rows, immediately flagging 2 missing values to investigate in the next topic.

11. Advantages

• Pandas' read_*() functions handle the vast majority of real-world file format quirks with minimal code.
• Immediate inspection tools (shape, head(), info()) make it easy to sanity-check a newly loaded dataset.

12. Limitations

• Very large files (many gigabytes) may require special handling (e.g., loading in chunks) since Pandas loads data into memory.
• Inconsistent file formatting (unusual delimiters, encodings, or malformed rows) can cause loading errors that require troubleshooting.

13. Common Mistakes

• Forgetting to check df.shape and df.head() immediately after loading, missing early clues about data issues.
• Assuming a CSV always uses commas as the delimiter — some files use semicolons, tabs, or other separators.
• Not specifying the correct sheet name when loading multi-sheet Excel files.
• Ignoring encoding errors instead of correctly specifying the encoding.

14. Best Practices

• Always inspect a dataset immediately after loading, before doing anything else.
• Explicitly specify parameters like sep, encoding, or sheet_name when the default doesn't work.
• Keep a copy of the original raw data file untouched, and work with a loaded DataFrame copy instead.

15. Real-World Applications

• Loading sales records, survey responses, sensor logs, or scientific datasets before any ML project.
• Loading data exported from business tools (Excel reports, database exports) into a Python-based ML workflow.

16. Interview-Oriented Points

• Be ready to name the correct Pandas function for loading CSV, Excel, and JSON files.
• Understand why immediately inspecting a newly loaded dataset (shape, head(), info()) is good practice.
• Be able to explain how you'd handle a CSV file that doesn't use commas as its delimiter.

17. Exam-Oriented Points

• pd.read_csv(), pd.read_excel(), and pd.read_json() are the primary functions for loading these respective formats.
• df.shape, df.head(), and df.info() are standard first steps after loading any dataset.
• The sep parameter in read_csv() specifies the delimiter if it's not a comma.

18. Comparison Table — CSV vs Excel vs JSON

19. Quick Revision

• pd.read_csv(), pd.read_excel(), and pd.read_json() load the most common data formats into a DataFrame.
• Always inspect a newly loaded dataset using shape, head(), and info().
• Watch out for delimiter, encoding, and header issues when loading files.
• Loading data correctly is the essential first step before any cleaning or modeling can happen.