Files

Whisker Jones aeba9bdb34 structure updates

2024-05-01 12:28:44 -06:00

3.0 KiB

Raw Blame History

For web scraping and working with HTML data in Python, Beautiful Soup is a highly useful library. It simplifies the process of extracting information from websites, allowing you to parse HTML and XML documents and navigate the parse tree or search for specific elements. Here's a concise reference guide for common use cases with Beautiful Soup, designed to help get you started with web scraping tasks:

`Beautiful Soup` Reference Guide

Installation

pip install beautifulsoup4

Note: You'll also need a parser like lxml or html.parser. For lxml, install it via pip install lxml.

Basic Usage

Importing Beautiful Soup

from bs4 import BeautifulSoup

Loading HTML Content

# Using a string
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')

# Using a web page content fetched with requests
import requests
response = requests.get('http://example.com')
soup = BeautifulSoup(response.content, 'html.parser')

Navigating the Parse Tree

Accessing Tags

# Access the title element
title_tag = soup.title

# Access the name of the tag
print(title_tag.name)

# Access the text within the title tag
print(title_tag.string)

Finding All Instances of a Tag

# Find all 'a' tags
a_tags = soup.find_all('a')

Accessing Attributes

# Access the first 'p' tag
p_tag = soup.find('p')

# Access its 'class' attribute
p_class = p_tag['class']

Searching the Tree

find and find_all

# Find the first 'a' tag
first_a_tag = soup.find('a')

# Find all 'a' tags with a specific class
specific_a_tags = soup.find_all('a', class_='sister')

# Find using a CSS selector
css_select_tags = soup.select('p.myclass')

Getting Data from Tags

# Get text content from a tag
for tag in soup.find_all('a'):
    print(tag.get_text())

# Get a specific attribute value
for tag in soup.find_all('a'):
    print(tag.get('href'))

Practical Example: Extracting Data from a Page

# Assuming you have fetched a page with product listings
for product in soup.find_all('div', class_='product'):
    name = product.h2.text
    price = product.find('span', class_='price').text
    print(f'Product: {name}, Price: {price}')

Beautiful Soup is designed to make your web scraping code more human-readable and concise. It's incredibly effective at parsing messy web HTML content and offers both simplicity and flexibility, making it accessible for beginners yet powerful enough for advanced users. This guide covers basic functionalities, but Beautiful Soup supports a wide array of parsing and navigating techniques, making it a versatile tool in your web scraping toolkit.

Beautiful Soup's flexibility and ease of use make it an excellent choice for both novice programmers and professional developers needing to extract information from the web, parse documents, or scrape data from websites efficiently.

3.0 KiB Raw Blame History

Beautiful Soup Reference Guide