Web Scraping

Extracting data from HTML with requests + BeautifulSoup — find()/find_all(), CSS selectors, pagination — plus when a static parser isn't enough and Selenium's real-browser automation is needed, and responsible/legal scraping practices.

Not every website offers a convenient API. Web scraping is the technique of extracting data directly from web pages — reading their HTML and pulling out exactly the information you need, programmatically.

1. What is Web Scraping?

What is it?

Web scraping means writing a program that automatically downloads and reads web pages, extracting specific data from their HTML structure.

Definition: Web scraping is the automated process of extracting data from websites by parsing their HTML content.

Why do we use it?

Collecting data (prices, reviews, listings) when no official API exists.
Monitoring changes on a page over time (price tracking, job listings).
Building datasets for research or analysis from publicly available web content.

2. HTML Basics (Quick Recap for Scraping)

What is it?

HTML (HyperText Markup Language) structures every webpage using tags — and scraping means reading this structure to find the data you want.

Simple Example

html
<html>
<body>
    <div class="product">
        <h2 class="title">Wireless Headphones</h2>
        <span class="price">$49.99</span>
    </div>
    <div class="product">
        <h2 class="title">Bluetooth Speaker</h2>
        <span class="price">$29.99</span>
    </div>
</body>
</html>

Important Points

Tags (<div>, <h2>, <span>) define structure.
class and id attributes are how scrapers (and CSS) target specific elements — this is exactly what you'll search for when scraping.

3. CSS Selectors (Quick Reference for Scraping)

What is it?

CSS selectors are patterns used to target specific HTML elements — the exact same syntax used to style web pages is also how scraping tools locate the data you want.

Selector	Matches
`div`	All `<div>` elements
`.classname`	Elements with `class="classname"`
`#idname`	The element with `id="idname"`
`div.product`	`<div>` elements with class `product`
`div .price`	Any element with class `price`, inside a `<div>`

4. `requests` + BeautifulSoup

What is it?

requests (covered in the APIs & HTTP file) downloads a webpage's raw HTML. BeautifulSoup then parses that HTML, letting you search and extract exactly the data you need.

bash
pip install requests beautifulsoup4

Simple Example

python
import requests
from bs4 import BeautifulSoup

html = """
<html>
<body>
    <div class="product">
        <h2 class="title">Wireless Headphones</h2>
        <span class="price">$49.99</span>
    </div>
    <div class="product">
        <h2 class="title">Bluetooth Speaker</h2>
        <span class="price">$29.99</span>
    </div>
</body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

products = soup.find_all("div", class_="product")

for product in products:
    title = product.find("h2", class_="title").text
    price = product.find("span", class_="price").text
    print(f"{title}: {price}")

Output:

Wireless Headphones: $49.99
Bluetooth Speaker: $29.99

Explanation of the Code

BeautifulSoup(html, "html.parser") parses the raw HTML into a navigable structure.
.find_all("div", class_="product") finds every matching element (returns a list). .find() finds only the first matching element.
.text extracts just the visible text content, stripping away the surrounding HTML tags.

Using CSS Selectors Directly

python
titles = soup.select("div.product .title")
for title in titles:
    print(title.text)

Explanation: .select() lets you use familiar CSS selector syntax directly, often more concise than chaining multiple .find() calls.

Scraping a Real Webpage

python
import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url, timeout=5)
soup = BeautifulSoup(response.text, "html.parser")

heading = soup.find("h1")
print(heading.text)

Extracting Attributes (like Links)

python
links = soup.find_all("a")
for link in links:
    print(link.get("href"))   # extracts the URL from href="..."

Common Mistakes

Forgetting websites change their HTML structure over time, potentially breaking a scraper that was working fine last week.
Not checking response.status_code before parsing — trying to parse an error page's HTML wastes time and produces wrong results.
Scraping data that changes dynamically via JavaScript after the initial page load — requests only gets the raw initial HTML, missing anything added afterward (see Selenium, below).

Important Points

.find() gets the first match; .find_all() gets every match, as a list.
.select() lets you use CSS selector syntax for more concise searches.

Practice

Given a block of sample HTML with 3 products (each with a title and price), extract and print all titles using find_all().

5. Handling Pagination

What is it?

Many websites split listings across multiple pages ("Page 1, 2, 3..."). Scraping the full dataset requires looping through each page.

Simple Example

python
import requests
from bs4 import BeautifulSoup

all_titles = []

for page in range(1, 4):   # scrape pages 1 to 3
    url = f"https://example.com/products?page={page}"
    response = requests.get(url, timeout=5)
    soup = BeautifulSoup(response.text, "html.parser")

    titles = soup.select(".title")
    for title in titles:
        all_titles.append(title.text)

print(all_titles)

Explanation: The loop constructs a new URL for each page number, following whatever URL pattern the target website uses for pagination (this varies from site to site — check the actual URL structure by browsing manually first).

Important Points

Always check what pattern a site uses for pagination before writing a scraper loop (?page=2, /page/2, etc. — it varies).
Add a short delay (time.sleep()) between requests to avoid overwhelming the server (see Responsible Scraping, below).

6. Dynamic Websites and Selenium

What is it?

Some websites load their content after the initial page load, using JavaScript (e.g., infinite-scroll pages, or data that only appears after clicking a button). requests + BeautifulSoup only sees the initial HTML — they can't run JavaScript. Selenium solves this by actually automating a real browser.

bash
pip install selenium

Simple Example

python
from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get("https://example.com")

time.sleep(2)   # give the page time to load JavaScript content

titles = driver.find_elements(By.CLASS_NAME, "title")
for title in titles:
    print(title.text)

driver.quit()

Explanation of the Code

webdriver.Chrome() opens an actual, automated Chrome browser window.
driver.get(url) navigates to the page, and — crucially — executes any JavaScript on it, just like a real user's browser would.
find_elements(By.CLASS_NAME, "title") searches the fully-rendered page (after JavaScript has run), unlike requests, which only ever sees the initial raw HTML.

Clicking Buttons and Waiting

python
button = driver.find_element(By.ID, "load-more")
button.click()
time.sleep(2)   # wait for new content to load after the click

Comparison Table — `requests`+BeautifulSoup vs Selenium

	`requests` + BeautifulSoup	Selenium
Runs JavaScript	No	Yes (real browser automation)
Speed	Fast	Slower (controls a real browser)
Resource usage	Light	Heavier
Best for	Static HTML pages	Dynamic, JavaScript-heavy pages

Common Mistakes

Using requests/BeautifulSoup on a JavaScript-heavy site and getting empty or incomplete results — a common source of scraping confusion.
Using time.sleep() with a fixed, arbitrary delay instead of properly waiting for a specific element to load (Selenium offers more robust "explicit wait" tools for production use).
Forgetting driver.quit(), leaving browser processes running in the background.

Important Points

Use requests + BeautifulSoup for simple, static pages — it's faster and lighter.
Use Selenium only when content genuinely requires JavaScript execution or interaction (clicking, scrolling) to appear.

7. Saving Scraped Results

What is it?

Once data is extracted, it's typically saved to a CSV or JSON file for later analysis (using the File Handling techniques from earlier in this course).

Simple Example

python
import csv

products = [
    {"title": "Wireless Headphones", "price": "$49.99"},
    {"title": "Bluetooth Speaker", "price": "$29.99"}
]

with open("products.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(products)

8. Responsible and Legal Scraping Practices

What is it?

Scraping isn't automatically legal or acceptable just because data is publicly visible — responsible scraping means respecting websites' rules and not overloading their servers.

Key Practices

Check `robots.txt` — most sites have a file at example.com/robots.txt stating which parts of the site scrapers are allowed (or not allowed) to access.
Read the Terms of Service — some sites explicitly prohibit scraping; ignoring this can have legal consequences.
Rate-limit your requests — add delays between requests (time.sleep()) so you don't overload the server or get your IP address blocked.
Identify yourself — some scrapers set a descriptive User-Agent header rather than pretending to be a regular browser.
Avoid scraping personal/private data — be especially careful with anything involving personal information, which may have legal privacy implications (like GDPR).
Prefer an official API when one exists — it's more stable, reliable, and explicitly permitted.

Important Points

Always check robots.txt and the site's Terms of Service before scraping.
Never hammer a server with rapid, unthrottled requests — this can be seen as a denial-of-service attack, and can get your IP address banned.

Common Beginner Mistakes — Summary for This Section

Not checking response.status_code before parsing.
Assuming requests can see JavaScript-rendered content — it can't; use Selenium for that.
Scraping too aggressively without delays, risking IP bans or server overload.
Ignoring robots.txt and a site's Terms of Service.

Cheat Sheet — Web Scraping

python
import requests
from bs4 import BeautifulSoup

response = requests.get(url, timeout=5)
soup = BeautifulSoup(response.text, "html.parser")

soup.find("div")                     # first match
soup.find_all("div", class_="name")   # all matches
soup.select("div.name .title")        # CSS selector syntax
element.text                           # visible text
element.get("href")                    # attribute value

# Selenium (for JavaScript-heavy sites)
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get(url)
driver.find_elements(By.CLASS_NAME, "name")
driver.quit()

Interview Questions

Q1. What is the difference between `requests`/BeautifulSoup and Selenium? Answer: requests/BeautifulSoup only downloads and parses the initial, static HTML of a page. Selenium automates a real browser, which can execute JavaScript and interact with dynamic content (clicking, scrolling) that only appears after the initial load.

Q2. What is `robots.txt`, and why does it matter for scraping? Answer: A file websites publish stating which parts of the site automated scrapers are allowed or disallowed to access — responsible scrapers check and respect it before scraping a site.

Q3. What is the difference between `.find()` and `.find_all()` in BeautifulSoup? Answer: .find() returns only the first matching element. .find_all() returns a list of every matching element.

Q4. Why is rate-limiting important when scraping? Answer: Sending requests too rapidly can overload the target server, degrade its performance for real users, and often results in your IP address being blocked.

Q5. When would you choose Selenium over `requests`/BeautifulSoup? Answer: When the needed content is loaded dynamically via JavaScript after the initial page load, or when the scraping process requires interacting with the page (clicking buttons, scrolling, filling forms).

Practice Questions

Beginner

Given a sample HTML string with 3 items in <li> tags, extract all of them using find_all().
Extract the text of the first <h1> tag from a sample HTML page.
Use .select() with a CSS selector to extract all elements with a specific class.
Extract all links (href values) from a sample HTML page containing several <a> tags.
Save a small list of scraped dictionaries to a CSV file.

Intermediate

Write a scraper that extracts product names and prices from a sample multi-product HTML page, and saves them to a CSV file.
Write a scraper that loops through 3 pages of a paginated sample URL structure, collecting all titles into one list.
Explain, in your own words (as a comment), why a scraper using requests might return incomplete data on a JavaScript-heavy website.
Write a Selenium script that opens a webpage, waits 2 seconds, and extracts all elements with a specific class name.
Check a real website's robots.txt file and summarize what it allows and disallows.

Challenge

Build a scraper that extracts book titles and prices from a public "scraping practice" website (many exist specifically for learning), handling pagination across multiple pages.
Write a Selenium script that clicks a "Load More" button on a sample dynamic page and then extracts the newly loaded content.
Design (in comments/pseudocode) a responsible scraping strategy for a hypothetical job-listings site, including rate-limiting, respecting robots.txt, and saving results incrementally to avoid data loss if the scraper crashes partway through.

1. What is Web Scraping?

What is it?

Why do we use it?

2. HTML Basics (Quick Recap for Scraping)

What is it?

Simple Example

Important Points

3. CSS Selectors (Quick Reference for Scraping)

What is it?

4. requests + BeautifulSoup

What is it?

Simple Example

Explanation of the Code

Using CSS Selectors Directly

Scraping a Real Webpage

Extracting Attributes (like Links)

Common Mistakes

Important Points

Practice

5. Handling Pagination

What is it?

Simple Example

Important Points

6. Dynamic Websites and Selenium

What is it?

Simple Example

Explanation of the Code

Clicking Buttons and Waiting

Comparison Table — requests+BeautifulSoup vs Selenium

Common Mistakes

Important Points

7. Saving Scraped Results

What is it?

Simple Example

8. Responsible and Legal Scraping Practices

What is it?

Key Practices

Important Points

Common Beginner Mistakes — Summary for This Section

Cheat Sheet — Web Scraping

Interview Questions

Practice Questions

Mock Test

Coding Problems

4. `requests` + BeautifulSoup

Comparison Table — `requests`+BeautifulSoup vs Selenium