Removing Duplicates

1. Introduction

Duplicate records — the same data appearing more than once — can silently distort your analysis and models, sometimes without any obvious warning sign. This short but important topic covers how to detect and remove duplicate rows from a dataset.

2. What are Duplicates?

Simple definition: Duplicates are rows in a dataset that are identical (or identical across specific important columns) to another row, effectively representing the same record more than once.

Technical explanation: A duplicate row is one whose values match another row's values exactly (across all columns, or a specified subset of columns), often arising from data entry errors, merging multiple data sources, or repeated data collection events.

3. Why is it Important?

• Duplicate records can artificially inflate certain patterns (e.g., making a customer's purchase count seem higher than reality), skewing analysis and model training.
• Undetected duplicates can lead a model to overemphasize certain data points simply because they appear multiple times.
• Removing duplicates is a standard, expected step in any thorough data cleaning process.

4. Prerequisites

Comfort with Pandas basics (Module 1, Topic 6).

5. Core Concepts

1. Detecting duplicate rows (duplicated())
2. Removing duplicate rows (drop_duplicates())
3. Full-row duplicates vs subset-based duplicates
4. Choosing which duplicate to keep (first, last)

6. Detailed Explanation

a) Detecting Duplicates

df.duplicated() returns True for each row that is an exact duplicate of a previous row, and False otherwise. Combined with .sum(), it gives a total count of duplicate rows.

b) Removing Duplicates

df.drop_duplicates() removes duplicate rows, keeping only the first occurrence by default.

c) Full-Row vs Subset-Based Duplicates

By default, a row must match every column's value to be considered a duplicate. Sometimes, you only care about duplicates based on specific columns (e.g., duplicate customer IDs, even if other details slightly differ) — this is done using the subset parameter.

d) Choosing Which Duplicate to Keep

The keep parameter controls which occurrence is retained: "first" (default), "last", or False (removes all occurrences of any duplicated row entirely).

7. How It Works

1. Check for duplicates using df.duplicated().sum() to understand the scope of the issue.
2. Decide whether duplicates should be judged across all columns or just specific key columns (e.g., a unique ID).
3. Apply df.drop_duplicates() with the appropriate subset and keep parameters.
4. Re-check df.duplicated().sum() to confirm duplicates have been removed, and verify the resulting dataset size makes sense.

8. Real-World Example

Imagine a customer database where the same customer accidentally submitted the same registration form twice due to a website glitch. Without removing this duplicate, that customer's information would be counted twice in any analysis — for example, incorrectly inflating the total customer count or over-representing that customer's preferences in a recommendation model.

9. Python Example

python
import pandas as pd

data = {
    "customer_id": [101, 102, 103, 101, 104],
    "name": ["Aarav", "Meera", "Kabir", "Aarav", "Diya"],
    "purchase_amount": [500, 700, 300, 500, 900]
}
df = pd.DataFrame(data)

print("Original data:")
print(df)

print("\nNumber of duplicate rows (full row match):", df.duplicated().sum())

# Removing full-row duplicates
df_cleaned = df.drop_duplicates()
print("\nAfter removing full-row duplicates:")
print(df_cleaned)

# Checking for duplicates based on customer_id only
print("\nDuplicates based on customer_id only:")
print(df.duplicated(subset=["customer_id"]).sum())

Expected Output:

text
Original data:
   customer_id   name  purchase_amount
0          101  Aarav              500
1          102  Meera              700
2          103  Kabir              300
3          101  Aarav              500
4          104   Diya              900

Number of duplicate rows (full row match): 1

After removing full-row duplicates:
   customer_id   name  purchase_amount
0          101  Aarav              500
1          102  Meera              700
2          103  Kabir              300
4          104   Diya              900

Duplicates based on customer_id only:
1

10. Code Explanation

• df.duplicated() compares each row to all previous rows across every column, flagging row index 3 as a duplicate of row index 0.
• .sum() converts the True/False results into a total count of duplicate rows.
• df.drop_duplicates() removes the duplicate row (index 3), keeping the first occurrence (index 0) by default.
• df.duplicated(subset=["customer_id"]) checks for duplicates based only on the customer_id column, which is useful when you specifically care about repeated IDs, regardless of whether other columns match exactly.

11. Advantages

• Prevents duplicate records from distorting analysis, statistics, or model training.
• Simple, fast, and well-supported directly in Pandas with just a couple of function calls.
• Subset-based duplicate detection offers flexibility for real-world scenarios where "duplicate" has a specific business meaning.

12. Limitations

• Near-duplicates (e.g., slightly different spellings of the same name) won't be caught by exact-match duplicate detection.
• Removing duplicates based on the wrong subset of columns can accidentally discard genuinely distinct records.
• Doesn't address duplicate information spread across differently structured records (e.g., duplicate content in different formats).

13. Common Mistakes

• Forgetting to check for duplicates at all before analysis or modeling.
• Using default full-row duplicate detection when subset-based detection (e.g., by unique ID) would be more meaningful.
• Removing duplicates without first understanding why they occurred, potentially masking a deeper data collection issue.

14. Best Practices

• Always check df.duplicated().sum() early in the cleaning process.
• Decide carefully whether full-row or subset-based duplicate detection is more appropriate for your specific dataset.
• Investigate why duplicates occurred if there are unexpectedly many — it may indicate a bug in the data collection process.

15. Real-World Applications

• Cleaning customer databases that may have accidental repeat registrations.
• Removing duplicate transaction records caused by system glitches or repeated form submissions.
• Deduplicating merged datasets combined from multiple sources.

16. Interview-Oriented Points

• Be ready to explain the difference between full-row duplicate detection and subset-based detection.
• Understand what the keep parameter in drop_duplicates() controls.
• Be able to explain a real-world scenario where duplicates could distort an ML model's results.

17. Exam-Oriented Points

• df.duplicated() detects duplicate rows; df.drop_duplicates() removes them.
• The subset parameter allows duplicate detection based on specific columns only.
• The keep parameter controls which occurrence (first, last, or none) is retained.

18. Comparison Table — Full-Row Duplicates vs Subset-Based Duplicates

19. Quick Revision

• df.duplicated() detects duplicate rows; df.drop_duplicates() removes them, keeping the first occurrence by default.
• Use the subset parameter to detect duplicates based on specific columns rather than the entire row.
• The keep parameter ("first", "last", False) controls which occurrence(s) are retained.
• Always check for and understand duplicates before proceeding with analysis or modeling.