Python Scrapy Library

Scrapy is a powerful Python web scraping framework specifically designed for crawling web page data and extracting information.

Scrapy is often used for data mining, information processing, or storing historical data and other applications.

Scrapy has many built-in useful features, such as handling requests, tracking states, handling errors, handling request rate limits, etc., making it very suitable for efficient, distributed web crawling.

Unlike simple scraping libraries (such as requests and BeautifulSoup), Scrapy is a full-featured scraping framework with high scalability and flexibility, suitable for complex and large-scale web scraping tasks.

Scrapy Official Website: https://scrapy.org/.

Scrapy Features and Introduction: https://www.runoob.com/w3cnote/scrapy-detail.html.

Scrapy Architecture Diagram (green lines represent data flow):

Scrapy’s work is based on the following core components:

Spider: Spider class, used to define how to extract data from web pages and how to follow links on the page.
Item: Used to define and store scraped data. Equivalent to a data model.
Pipeline: Used to process scraped data, commonly used for cleaning, storing data, and other operations.
Middleware: Used to handle requests and responses, can be used to set proxies, handle cookies, user agents, etc.
Settings: Used to configure various settings of the Scrapy project, such as request delay, concurrent request count, etc.

Installing Scrapy

Before using Scrapy, you need to install it first. Use pip to install:

pip install scrapy

Scrapy Project Structure

A Scrapy project is a structured directory containing multiple folders and modules designed to help you organize your spider code.

Scrapy uses command-line tools to create and manage spider projects. You can use the following command to create a new Scrapy project:

scrapy startproject myproject

This will create a project named myproject with the following project structure:

myproject/
    scrapy.cfg            # Project configuration file
    myproject/            # Project source code folder
        __init__.py
        items.py          # Define the data structure for scraping
        middlewares.py    # Define middlewares
        pipelines.py      # Define data processing pipelines
        settings.py       # Project settings file
        spiders/           # Folder for storing spider code
            __init__.py
            myspider.py   # Custom spider code

Writing a Simple Scrapy Spider

The following is a basic Scrapy spider example demonstrating how to scrape data from a web page.

Let’s create a spider project:

scrapy startproject runoob_test_spiders

Executing the above command, if successful, will output:

templates/project', created in:/Users/Runoob/runoob-test/runoob_test_spiders

You can start your first spider with:
    cd runoob_test_spiders
    scrapy genspider example example.com

The generated project structure is as follows:

Then enter the directory:

cd runoob_test_spiders

Next, create a spider using the scrapy genspider command:

scrapy genspider douban_spider movie.douban.com

The directory structure is as follows:

A file named douban_spider.py is generated under the runoob_test_spiders directory with the following code:

Example

import scrapy

class DoubanSpiderSpider(scrapy.Spider):
    name = "douban_spider"
    allowed_domains = ["movie.douban.com"]
    start_urls = ["https://movie.douban.com"]

    def parse(self, response):
        pass

Code Explanation:

name: Defines the spider’s name, must be unique.
allowed_domains: Restricts the spider’s access domain, preventing the spider from crawling pages on other domains.
start_urls: Defines the spider’s starting pages. The spider will begin crawling from these pages.
parse: The parse method is the core part of each spider, used to process the response and extract data. It receives a response object representing the page content returned by the server.

Writing Spider Code

Before writing spider code, we need to note a few points:

Websites like Douban may detect spider behavior. It is recommended to set USER_AGENT and DOWNLOAD_DELAY to simulate normal user behavior.
When scraping data, please comply with the target website’s robots.txt file regulations and avoid putting excessive pressure on the server.
Frequent crawling may trigger IP bans.

Modifying settings.py Configuration

Add the following configuration in settings.py to simulate browser requests and bypass anti-scraping mechanisms:

# Set User-Agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'

# Do not comply with robots.txt rules
ROBOTSTXT_OBEY = False

# Set download delay to avoid excessively fast requests
DOWNLOAD_DELAY = 2

# Enable automatic throttling extension
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2
AUTOTHROTTLE_MAX_DELAY = 5

In the spider code, add custom request headers (such as User-Agent and Referer) to further simulate browser behavior.

Open the douban_spider.py file and modify its content as follows:

Example

import scrapy

class DoubanSpider(scrapy.Spider):
    name = "douban_spider"
    start_urls = [
        'https://movie.douban.com/top250',
    ]

    def start_requests(self):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
            'Referer': 'https://movie.douban.com/',
        }
        for url in self.start_urls:
            yield scrapy.Request(url, headers=headers, callback=self.parse)

    def parse(self, response):
        for movie in response.css('div.item'):
            yield {
                'title': movie.css('span.title::text').get(),
                'rating': movie.css('span.rating_num::text').get(),
                'quote': movie.css('span.inq::text').get(),
            }

        # Handle pagination
        next_page = response.css('span.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

Code Analysis:

name = "douban_spider": Defines the spider’s name.
start_urls: Defines the initial URL where the spider starts crawling (Douban Movie Top 250 page).
parse Method:
- Uses CSS selectors to extract the title, rating, and quote of each movie.
- span.title::text: Extracts the movie title.
- span.rating_num::text: Extracts the movie rating.
- span.inq::text: Extracts the movie quote.
Pagination Handling:
- Uses span.next a::attr(href) to extract the link to the next page.
- If a next page exists, uses response.follow to continue crawling.

Run the following command in the command line to start the spider:

scrapy crawl douban_spider -o douban_movies.csv

This will start the spider and save the extracted data to the douban_movies.csv file.

Note: The above content is for learning purposes only. When scraping data, please comply with the target website’s robots.txt file regulations.

Common Methods

1. Spider Methods

Method Name	Description	Example
`start_requests()`	Generate initial requests. Can customize request headers, methods, etc.	`yield scrapy.Request(url, callback=self.parse)`
`parse(response)`	Process the response and extract data. The core method of the spider.	`yield {'title': response.css('h1::text').get()}`
`follow(url, callback)`	Automatically handle relative URLs and generate new requests, used for pagination or link following.	`yield response.follow(next_page, callback=self.parse)`
`closed(reason)`	Called when the spider is closed, used for cleaning up resources or logging.	`def closed(self, reason): print('Spider closed:', reason)`
`log(message)`	Log a message.	`self.log('This is a log message')`

2. Data Extraction Methods

Method Name	Description	Example
`response.css(selector)`	Extract data using CSS selectors.	`title = response.css('h1::text').get()`
`response.xpath(selector)`	Extract data using XPath selectors.	`title = response.xpath('//h1/text()').get()`
`get()`	Extract the first matching result from `SelectorList` (string).	`title = response.css('h1::text').get()`
`getall()`	Extract all matching results from `SelectorList` (list).	`titles = response.css('h1::text').getall()`
`attrib`	Extract the attribute of the current node.	`link = response.css('a::attr(href)').get()`

3. Request and Response Methods

Method Name	Description	Example
`scrapy.Request(url, callback, method, headers, meta)`	Create a new request.	`yield scrapy.Request(url, callback=self.parse, headers=headers)`
`response.url`	Get the URL of the current response.	`current_url = response.url`
`response.status`	Get the status code of the response.	`if response.status == 200: print('Success')`
`response.meta`	Get additional data passed in the request.	`value = response.meta.get('key')`
`response.headers`	Get the header information of the response.	`content_type = response.headers.get('Content-Type')`

4. Middleware and Pipeline Methods

Method Name	Description	Example
`process_request(request, spider)`	Process the request before it is sent (downloader middleware).	`request.headers['User-Agent'] = 'Mozilla/5.0'`
`process_response(request, response, spider)`	Process the response after it is returned (downloader middleware).	`if response.status == 403: return request.replace(dont_filter=True)`
`process_item(item, spider)`	Process the extracted data (pipeline).	`if item['price'] < 0: raise DropItem('Invalid price')`
`open_spider(spider)`	Called when the spider starts (pipeline).	`def open_spider(self, spider): self.file = open('items.json', 'w')`
`close_spider(spider)`	Called when the spider is closed (pipeline).	`def close_spider(self, spider): self.file.close()`

5. Tools and Extension Methods

Method Name	Description	Example
`scrapy shell`	Launch an interactive shell for debugging and testing selectors.	`scrapy shell 'http://example.com'`
`scrapy crawl <spider_name>`	Run the specified spider.	`scrapy crawl myspider -o output.json`
`scrapy check`	Check the correctness of the spider code.	`scrapy check`
`scrapy fetch`	Download the content of the specified URL.	`scrapy fetch 'http://example.com'`
`scrapy view`	View the page downloaded by Scrapy in a browser.	`scrapy view 'http://example.com'`

6. Common Settings (`settings.py`)

Setting Item	Description	Example
`USER_AGENT`	Set the User-Agent in the request header.	`USER_AGENT = 'Mozilla/5.0'`
`ROBOTSTXT_OBEY`	Whether to comply with `robots.txt` rules.	`ROBOTSTXT_OBEY = False`
`DOWNLOAD_DELAY`	Set download delay to avoid excessively fast requests.	`DOWNLOAD_DELAY = 2`
`CONCURRENT_REQUESTS`	Set the number of concurrent requests.	`CONCURRENT_REQUESTS = 16`
`ITEM_PIPELINES`	Enable pipelines.	`ITEM_PIPELINES = {'myproject.pipelines.MyPipeline': 300}`
`AUTOTHROTTLE_ENABLED`	Enable automatic throttling extension.	`AUTOTHROTTLE_ENABLED = True`

7. Other Common Methods

Method Name	Description	Example
`response.follow_all(links, callback)`	Batch process links and generate requests.	`yield from response.follow_all(links, callback=self.parse)`
`response.json()`	Parse the response content as JSON format.	`data = response.json()`
`response.text`	Get the text content of the response.	`html = response.text`
`response.selector`	Get the `Selector` object of the response content.	`title = response.selector.css('h1::text').get()`

The above tables list the commonly used methods in Scrapy and their functions. These methods cover various aspects of spider development, including request generation, data extraction, middleware processing, pipeline operations, etc. By mastering these methods, you can efficiently write and manage Scrapy spiders. If you need more detailed features, refer to the Scrapy Official Documentation.

Method Usage Examples

1. `start_requests()`

The start_requests() method is the entry point of a Scrapy spider, used to generate initial requests. Typically, the spider’s starting URLs are defined in this method.

Example

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'
    
    def start_requests(self):
        urls = [
            'http://example.com/page1',
            'http://example.com/page2',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

2. `parse()`

The parse() method is the default response processing method, used to parse the response and extract data or generate new requests.

Example

def parse(self, response):
    # Extract the page title
    title = response.css('title::text').get()
    yield {
        'title': title
    }

3. `parse_item()`

The parse_item() method is used to parse the response of a single item, typically used to extract structured data.

Example

def parse_item(self, response):
    item = {}
    item['name'] = response.css('div.name::text').get()
    item['price'] = response.css('div.price::text').get()
    yield item

4. `follow()`

The follow() method is used to generate new requests and automatically process the response, typically used for following links.

Example

def parse(self, response):
    for link in response.css('a::attr(href)'):
        yield response.follow(link, self.parse_item)

5. `yield`

The yield keyword is used to generate requests or items and pass them to the Scrapy engine for processing.

Example

def parse(self, response):
    yield {
        'title': response.css('title::text').get()
    }

6. `Item`

The Item class is used to define data structures, typically used to store data extracted from web pages.

Example

import scrapy

class MyItem(scrapy.Item):
    name = scrapy.Field()
    price = scrapy.Field()

7. `ItemLoader`

The ItemLoader class is used to load and populate Item objects, simplifying the data extraction and processing workflow.

Example

from scrapy.loader import ItemLoader
from myproject.items import MyItem

def parse(self, response):
    loader = ItemLoader(item=MyItem(), response=response)
    loader.add_css('name', 'div.name::text')
    loader.add_css('price', 'div.price::text')
    yield loader.load_item()

8. `Request`

The Request class is used to generate HTTP request objects, typically used to define the request’s URL, callback method, etc.

Example

import scrapy

def parse(self, response):
    yield scrapy.Request(url='http://example.com/page3', callback=self.parse_item)

9. `Response`

The Response class represents an HTTP response object, containing HTML content, status code, and other information returned from the server.

Example

def parse(self, response):
    print(response.status)  # Print the response status code
    print(response.body)    # Print the response content

10. `Selector`

The Selector class is used to extract data from HTML or XML documents, supporting XPath and CSS selectors.

Example

def parse(self, response):
    title = response.xpath('//title/text()').get()
    yield {
        'title': title
    }

11. `CrawlSpider`

CrawlSpider is a special Spider class used to handle complex crawling rules and link following.

Example

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MyCrawlSpider(CrawlSpider):
    name = 'mycrawlspider'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com']

    rules = (
        Rule(LinkExtractor(allow=('page/\d+',)), callback='parse_item'),
    )

    def parse_item(self, response):
        yield {
            'title': response.css('title::text').get()
        }

12. `LinkExtractor`

The LinkExtractor class is used to extract links from responses, typically used for automatically following links on a page.

Example

from scrapy.linkextractors import LinkExtractor

def parse(self, response):
    extractor = LinkExtractor(allow=('page/\d+',))
    links = extractor.extract_links(response)
    for link in links:
        yield scrapy.Request(link.url, callback=self.parse_item)

13. `Pipeline`

The Pipeline class is used to process scraped data, typically used for data cleaning, storage, and other operations.

Example

class MyPipeline:
    def process_item(self, item, spider):
        # Process item data
        return item

14. `Middleware`

The Middleware class is used to process requests and responses, typically used for modifying request headers, handling exceptions, and other operations.

Example

class MyMiddleware:
    def process_request(self, request, spider):
        # Modify the request header
        request.headers['User-Agent'] = 'MyCustomUserAgent'
        return None

Python Scrapy Library

Installing Scrapy

Scrapy Project Structure

Writing a Simple Scrapy Spider

Example

Writing Spider Code

Modifying settings.py Configuration

Example

Common Methods

1. Spider Methods

2. Data Extraction Methods

3. Request and Response Methods

4. Middleware and Pipeline Methods

5. Tools and Extension Methods

6. Common Settings (settings.py)

7. Other Common Methods

Method Usage Examples

1. start_requests()

Example

2. parse()

Example

3. parse_item()

Example

4. follow()

Example

5. yield

Example

6. Item

Example

7. ItemLoader

Example

8. Request

Example

9. Response

Example

10. Selector

Example

11. CrawlSpider

Example

12. LinkExtractor

Example

13. Pipeline

Example

14. Middleware

Example

6. Common Settings (`settings.py`)

1. `start_requests()`

2. `parse()`

3. `parse_item()`

4. `follow()`

5. `yield`

6. `Item`

7. `ItemLoader`

8. `Request`

9. `Response`

10. `Selector`

11. `CrawlSpider`

12. `LinkExtractor`

13. `Pipeline`

14. `Middleware`