Python Web Scraping – BeautifulSoup

Python web scraping refers to the process of automatically extracting information from the internet by writing Python programs.

The basic process of web scraping typically includes sending HTTP requests to obtain web page content, parsing the web page and extracting data, and then storing the data.

Python’s rich ecosystem makes it a popular language for developing web scrapers, especially due to its powerful library support.

Generally, the web scraping process can be divided into the following steps:

• Send HTTP Requests: The scraper obtains the HTML page from the target website via HTTP requests. Common libraries include requests.
• Parse HTML Content: After obtaining the HTML page, the scraper needs to parse the content and extract data. Common libraries include BeautifulSoup, lxml, Scrapy, etc.
• Extract Data: Locate HTML elements (such as tags, attributes, class names, etc.) to extract the required data.
• Store Data: Store the extracted data in databases, CSV files, JSON files, and other formats for subsequent use or analysis.

This chapter mainly introduces BeautifulSoup, which is a Python library for parsing HTML and XML documents, capable of extracting data from web pages, commonly used for web scraping and data mining.

---

BeautifulSoup

BeautifulSoup is a Python library for extracting data from web pages, particularly suitable for parsing HTML and XML files.

BeautifulSoup can extract and manipulate content from web pages by providing a simple API, making it very suitable for web scraping and data extraction tasks.

Installing BeautifulSoup

To use BeautifulSoup, you need to install beautifulsoup4 and lxml or html.parser (an HTML parser).

You can use pip to install these dependencies:

pip install beautifulsoup4
pip install lxml  # It is recommended to use lxml as the parser (faster)

If you don’t have lxml, you can use Python’s built-in html.parser as the parser.

Basic Usage

BeautifulSoup is used to parse HTML or XML data and provides methods for navigating, searching, and modifying the parse tree.

Common operations of BeautifulSoup include finding tags, getting tag attributes, extracting text, etc.

To use BeautifulSoup, you need to import BeautifulSoup first and load the HTML page into a BeautifulSoup object.

Typically, you will first use a scraping library (such as requests) to get the web page content:

Example

from bs4 import BeautifulSoup
import requests

# Use requests to get web page content
url = 'https://cn.bing.com/' # Scrape the web page content of the Bing search engine
response = requests.get(url)

# Use BeautifulSoup to parse the web page
soup = BeautifulSoup(response.text, 'lxml')  # Use lxml parser
# Parse web page content with html.parser
# soup = BeautifulSoup(response.text, 'html.parser')

Get the web page title:

Example

from bs4 import BeautifulSoup
import requests

# Specify the website you want to get the title from
url = 'https://cn.bing.com/' # Scrape the web page content of the Bing search engine

# Send HTTP request to get web page content
response = requests.get(url)
# Chinese garbled character issue
response.encoding = 'utf-8' 
# Ensure the request was successful
if response.status_code == 200:
    # Use BeautifulSoup to parse web page content
    soup = BeautifulSoup(response.text, 'lxml')
    
    # Find the <title> tag
    title_tag = soup.find('title')
    
    # Print the title text
    if title_tag:
        print(title_tag.get_text())
    else:
        print("No <title> tag found")
else:
    print("Request failed, status code:", response.status_code)

Executing the above code outputs the title:

Search - Microsoft Bing

Chinese Encoding Issues

When using the requests library to scrape Chinese web pages, you may encounter encoding issues that prevent Chinese content from being displayed correctly. To ensure correct scraping and display of Chinese web pages, you typically need to handle the web page’s character encoding.

Auto-detecting encoding: requests usually automatically infers the web page’s encoding based on the Content-Type in the response headers, but sometimes it may be inaccurate. In such cases, you can use chardet to auto-detect the encoding.

Example

import requests

url = 'https://cn.bing.com/'
response = requests.get(url)

# Use chardet to auto-detect encoding
import chardet
encoding = chardet.detect(response.content)['encoding']
print(encoding)
response.encoding = encoding

Executing the above code outputs:

utf-8

If you know the web page’s encoding (e.g., utf-8 or gbk), you can directly set response.encoding:

response.encoding = 'utf-8'  # Or 'gbk', choose based on the actual situation

Finding Tags

BeautifulSoup provides various methods to find tags in web pages. The most commonly used ones include find() and find_all().

• find() returns the first matching tag
• find_all() returns all matching tags

Example

from bs4 import BeautifulSoup
import requests

# Specify the website you want to get the title from
url = 'https://www.baidu.com/' # Scrape the web page content of the Baidu search engine

# Send HTTP request to get web page content
response = requests.get(url)
# Chinese garbled character issue
response.encoding = 'utf-8'

soup = BeautifulSoup(response.text, 'lxml')

# Find the first <a> tag
first_link = soup.find('a')
print(first_link)
print("----------------------------")

# Get the href attribute of the first <a> tag
first_link_url = first_link.get('href')
print(first_link_url)
print("----------------------------")

# Find all <a> tags
all_links = soup.find_all('a')
print(all_links)

The output result is similar to the following:

<a class="mnav" href="http://news.baidu.com" name="tj_trnews">News</a>
----------------------------
http://news.baidu.com
----------------------------
[<a class="mnav" href="http://news.baidu.com" name="tj_trnews">News</a>, <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a>, <a class="mnav" href="http://map.baidu.com" name="tj_trmap">Maps</a>, <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">Videos</a>,

Getting Tag Text

Through the get_text() method, you can extract the text content in a tag:

Example

from bs4 import BeautifulSoup
import requests

# Specify the website you want to get the title from
url = 'https://www.baidu.com/' # Scrape the web page content of the Baidu search engine

# Send HTTP request to get web page content
response = requests.get(url)
# Chinese garbled character issue
response.encoding = 'utf-8'

soup = BeautifulSoup(response.text, 'lxml')

# Get the text content in the first <p> tag
paragraph_text = soup.find('p').get_text()

# Get all text content on the page
all_text = soup.get_text()
print(all_text)

The output result is similar to the following:

 Baidu, just search it           
...

Finding Child Tags and Parent Tags

You can access a tag’s parent tag and child tags through the parent and children attributes:

# Get the parent tag of the current tag
parent_tag = first_link.parent

# Get all child tags of the current tag
children = first_link.children

Example

from bs4 import BeautifulSoup
import requests

# Specify the website you want to get the title from
url = 'https://www.baidu.com/' # Scrape the web page content of the Baidu search engine

# Send HTTP request to get web page content
response = requests.get(url)
# Chinese garbled character issue
response.encoding = 'utf-8'

soup = BeautifulSoup(response.text, 'lxml')

# Find the first <a> tag
first_link = soup.find('a')
print(first_link)
print("----------------------------")

# Get the parent tag of the current tag
parent_tag = first_link.parent
print(parent_tag.get_text())

The output result is similar to the following:

<a class="mnav" href="http://news.baidu.com" name="tj_trnews">News</a>
----------------------------
 News hao123 Maps Videos Tieba Login More Products 

Finding Tags with Specific Attributes

You can find tags with specific attributes by passing attributes.

For example, find all div tags with the class name example-class:

# Find all <div> tags with class="example-class"
divs_with_class = soup.find_all('div', class_='example-class')

# Find the <p> tag with id="unique-id"
unique_paragraph = soup.find('p', id='unique-id')

Get the search button with id su:

Example

from bs4 import BeautifulSoup
import requests

# Specify the website you want to get the title from
url = 'https://www.baidu.com/' # Scrape the web page content of the Baidu search engine

# Send HTTP request to get web page content
response = requests.get(url)
# Chinese garbled character issue
response.encoding = 'utf-8'

soup = BeautifulSoup(response.text, 'lxml')

# Find the <input> tag with id="unique-id"
unique_input = soup.find('input', id='su')

input_value = unique_input['value'] # Get the value of the input field

print(input_value)

The output result is:

Baidu Search

---

Advanced Usage

CSS Selectors

BeautifulSoup also supports finding tags through CSS selectors.

The select() method allows using jQuery-like selector syntax to find tags:

# Use CSS selector to find all <div> tags with class 'example'
example_divs = soup.select('div.example')

# Find all href attributes in <a> tags
links = soup.select('a[href]')

Handling Nested Tags

BeautifulSoup supports deeply nested HTML structures. You can handle these structures by recursively finding child tags:

# Find nested <div> tags
nested_divs = soup.find_all('div', class_='nested')
for div in nested_divs:
    print(div.get_text())

Modifying Web Page Content

BeautifulSoup allows you to modify HTML content.

You can modify tag attributes, text, or delete tags:

Example

# Modify the href attribute of the first <a> tag
first_link['href'] = 'http://new-url.com'

# Modify the text content of the first <p> tag
first_paragraph = soup.find('p')
first_paragraph.string = 'Updated content'

# Delete a tag
first_paragraph.decompose()

Converting to String

You can convert the parsed BeautifulSoup object back to an HTML string:

# Convert to string
html_str = str(soup)

---

BeautifulSoup Attributes and Methods

The following are commonly used attributes and methods in BeautifulSoup:

Method/Attribute	Description	Example
BeautifulSoup()	Parse HTML or XML documents and return a BeautifulSoup object.	soup = BeautifulSoup(html_doc, 'html.parser')
.prettify()	Format and beautify document content, generating structured strings.	print(soup.prettify())
.find()	Find the first matching tag.	tag = soup.find('a')
.find_all()	Find all matching tags, returning a list.	tags = soup.find_all('a')
.find_all_next()	Find all tags after the current tag that match the criteria.	tags = soup.find('div').find_all_next('p')
.find_all_previous()	Find all tags before the current tag that match the criteria.	tags = soup.find('div').find_all_previous('p')
.find_parent()	Return the parent tag of the current tag.	parent = tag.find_parent()
.find_all_parents()	Find all parent tags of the current tag.	parents = tag.find_all_parents()
.find_next_sibling()	Find the next sibling tag of the current tag.	next_sibling = tag.find_next_sibling()
.find_previous_sibling()	Find the previous sibling tag of the current tag.	prev_sibling = tag.find_previous_sibling()
.parent	Get the parent tag of the current tag.	parent = tag.parent
.next_sibling	Get the next sibling tag of the current tag.	next_sibling = tag.next_sibling
.previous_sibling	Get the previous sibling tag of the current tag.	prev_sibling = tag.previous_sibling
.get_text()	Extract the text content within a tag, ignoring all HTML tags.	text = tag.get_text()
.attrs	Return all attributes of a tag, represented as a dictionary.	href = tag.attrs['href']
.string	Get the string content within a tag.	string_content = tag.string
.name	Return the name of the tag.	tag_name = tag.name
.contents	Return all child elements of a tag, as a list.	children = tag.contents
.descendants	Return all descendant elements of a tag, as a generator.	for child in tag.descendants: print(child)
.parent	Get the parent tag of the current tag.	parent = tag.parent
.previous_element	Get the previous element of the current tag (excluding text).	prev_elem = tag.previous_element
.next_element	Get the next element of the current tag (excluding text).	next_elem = tag.next_element
.decompose()	Remove the current tag and its content from the tree.	tag.decompose()
.unwrap()	Remove the tag itself, keeping only its child content.	tag.unwrap()
.insert()	Insert a new tag or text into the tag.	tag.insert(0, new_tag)
.insert_before()	Insert a new tag before the current tag.	tag.insert_before(new_tag)
.insert_after()	Insert a new tag after the current tag.	tag.insert_after(new_tag)
.extract()	Remove the tag and return it.	extracted_tag = tag.extract()
.replace_with()	Replace the current tag and its content.	tag.replace_with(new_tag)
.has_attr()	Check if the tag has a specified attribute.	if tag.has_attr('href'):
.get()	Get the value of a specified attribute.	href = tag.get('href')
.clear()	Clear all content of the tag.	tag.clear()
.encode()	Encode the tag content as a byte stream.	encoded = tag.encode()
.is_empty_element	Check if the tag is an empty element (e.g., <br>, <img>, etc.).	if tag.is_empty_element:
.is_ancestor_of()	Check if the current tag is an ancestor of the specified tag.	if tag.is_ancestor_of(another_tag):
.is_descendant_of()	Check if the current tag is a descendant of the specified tag.	if tag.is_descendant_of(another_tag):

Other Attributes

Method/Attribute	Description	Example
.style	Get the inline style of the tag.	style = tag['style']
.id	Get the id attribute of the tag.	id = tag['id']
.class_	Get the class attribute of the tag.	class_name = tag['class']
.string	Get the string content within the tag, ignoring other tags.	content = tag.string
.parent	Get the parent element of the tag.	parent = tag.parent

Others

Method/Attribute	Description	Example
find_all(string)	Use strings to find matching tags.	tag = soup.find_all('div', class_='container')
find_all(id)	Find tags with the specified id.	tag = soup.find_all(id='main')
find_all(attrs)	Find tags with specified attributes.	tag = soup.find_all(attrs={"href": "http://example.com"})