TDM 20200: Project 3 - Web Scraping Introduction 3

Let’s start by learning about rate limiting with simple delays. First, these are the imports used.

We start with a function that retrieves a website. It takes in url, which is the website to be scraped, and delay, which the number of seconds to wait after the initial request. We will get more into that soon.

As we can see by the comment, requests.get(url) sends the HTTP GET request (trying to get data of a source, such as a website here), and eventually server sends us the response back, contianing information such as status code, metadata, and HTML content. These information gets put into a Response object by Python, and for us, in 'response' as we defined.

Below, we create a list to store the page URLs to scrape.

After this, the different URLs will look like: - quotes.toscrape.com/page/1/, quotes.toscrape.com/page/2/, and quotes.toscrape.com/page/3/

Now we move onto the main data collection loop.

It is a natural to ask what this expression represents: '//span[@class="text"]/text()'. Let’s take a closer look.

Now, implement a scraper that:
1. Scrapes multiple pages from quotes.toscrape.com/
2. Adds a 2-second delay between each request,
3. Tracks how long the scraping takes,
4. Calculates the average time per request.

When a server is overloaded or you’re making requests too quickly, you might receive HTTP error codes like 429 (Too Many Requests) or 503 (Service Unavailable). Exponential backoff is a strategy where you wait progressively longer between retries when you encounter errors.

As the name might suggest, the above import statement generates random numbers.

We now create a function that incorporates automatic retries in requesting a webpage.

HTTP Status Codes are used here. For instace, response.status_code == 429 checks to see if there are too many requests.

You might also notice the try/except used here; in Python, we use them to handle errors effectively. try block executes some code, also testing for errors, and except block is for handling the error.

except requests.exceptions.RequestException as e: Our code uses this to catch all request library related errors, such as connection and timeouts. 'e' is just a variable we assign this exception instance to, that holds the specific error message and related details.

Now, implement a scraper with exponential backoff that:
1. Handles 429 (Too Many Requests) errors,
2. Handles 503 (Service Unavailable) errors,
3. Implements exponential backoff with jitter,
4. Logs retry attempts and wait times,
5. Scrapes multiple pages, handling errors gracefully.

Some websites check the "User-Agent" header to identify what type of browser or program is making the request. Some sites block requests that don’t have a proper User-Agent or that look like bots. We can customize our request headers to appear more like a regular browser:

User agent strings contain information such as browser name, version, and operating system, in a text line which website use to get information on the client making the request.

We recommend you to read more about it online if you’re interested (some sites, such as 51degrees.com/blog/understanding-user-agent-string, have some good summary to gain more understanding)

First, we create a function to send HTTP request. It takes in two parameters: url specifying the page to scrape, and an optional user_agent . If we don’t specify, the default is used

Also note that the quality value is included, as seen in q=0.9,/;q=0.8', where the higher number denotes higher preference (priority). The range is 0.0 to 1.0. So, for example in our code / has a quality value of 0.8, it has lower priorty compared to things like text/html (1.0), application/xhtml+xml (1.0), and application/xml (0.9).

Similarly as above, q denotes the quality value, specifically secondary preference. In our code, we accept US english as primary preference, and other English types with lower priority.

For further information: developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language

'gzip' and 'deflate' are compression formats (popular options). We can accept either type of responses.

No compression can be applied by specifying 'identity'. A general matching can be done using a 'wildcard (*)', which accepts any encoding even if it was not given.

You can read more about other types and what they are here: developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language

By default, HTTP/1.0 closes the connection after a request.

For further information: developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Keep-Alive

Below is an example of scraping with custom headers

Some websites also use cookies for session management. We can handle cookies using a session object:

A session can remember cookies, share headers, and reuse connections. Cookies and headers would not be remembered between requests in each requests.get() without this.

Documentation: requests.readthedocs.io/en/latest/user/advanced/

User-Agent is the header name, and we specified the value of 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36', assigning a browser user agent to the session. It gets applied to all request made by session.

In our code, we first send the HTTP GET request to url1’s URL. Then, through the session object, it gets headers, cookies, and maintains the connection settings. Response object (response variable) containing HTML content, HTTP status code, and cookies (session.cookies)

Now, create a scraper that:
1. Uses custom User-Agent headers,
2. Maintains a session across multiple requests,
3. Rotates between different User-Agent strings for different requests,
4. Scrapes multiple pages while maintaining the session.

In this question, we explore some rules that should be followed before scraping websites, and how to incorporate them into our code. More specifically, we need to follow the instructions given in a file called robots.txt in websites. So in Question 4, we aim to create a function that checks the robots.txt rules and adheres to them by only accessing the allowed paths.

We have two new imports:

First Import:

RobotFileParser class becomes available to us with this import statement. This is what provides us with the Python’s built in robots.txt file, a standard that websites use to tell scrapers which parts of the site they can and cannot access.

robots.txt is an implementation of the Robots Exclusion Protocol, which defines set of rules for web crawlers and web robots on what sections of a website they may or may not visit. The text file is located at the root of a website.

Couple notes on this: If this file can not be found, the assumption is that there is no restriction on accessing the whole website. As well, if there are multiple subdomains in a website, they need to have their individual 'robots.txt' file.

Second Import:

urllib.parse is a Python module that provides us with various functions that can work with and manipulate URLs. For instance,

1) Break up URLs which initially is one string, into different componenets such as addressing scheme, network location, path, etc

This is where urlparse comes in. urlparse breaks a single string down into six components, and returns ParseResult(scheme, netloc, path, params, query, fragment). As you might be able to tell from its parameters, this gives us the ability to easily access specific parts of the URL.

Here is a simple example. If we had: base_url = "https://parts.toscrape.com/page/1/", then parsed_url = urlparse(base_url) gives us:

scheme = "https", netloc = "parts.toscrape.com", and path = "/page/1/". params, query, and framgent are "".

The last three are empty because the URL simply does not contain them. If they exist, you will see 'query' after '?' and 'fragment' after '#'. Some other parameters include '=', which is for key-value pair (key=value), or '&' to separate multiple parameters.

2) Convert relative URL into absolute URL. urljoin is used here.

urljoin takes in two arguments, 'base' and 'url', where if given the relative URL, the function combines it with the 'base' URL to create the absolute version.

Now, let’s get started on creating a function that checks a website’s 'robots.txt' rules.

It takes in two parameters, 'base_url' is the website to scrape, and 'user_agent='*'' is to specify which crawler to check rules for. Similarly as our other cases where * was used, here it means rules that apply for all bots.

Please see below for more explanation!

As initially mentioned, this is how crawlers know when they can or can not access. This is done through the 'Disallow' and 'Allow' directives. 'robots.txt' generally has the structure of:

User-agent: *
Disallow:
Allow:

'Disallow' is an instruction to not access a path or a directory. There usually is a path that specifies the location. For example, if we had 'Disallow: /admin/', it corresponds to not allowing crawlers to visit 'https://example.com/admin/'. Having an empty 'Disallow' is a permission to access everything; 'Allow' can be used to allow specific path access, although the empty 'Disallow' is a broader permission.

Now, create a function that:
1. Checks robots.txt before scraping,
2. Respects crawl delays specified in robots.txt,
3. Only scrapes paths that are allowed,
4. Implements a scraper that checks multiple websites' robots.txt files.

Test with:
- quotes.toscrape.com/
- books.toscrape.com/
- www.google.com/ (to see a more complex robots.txt)

So far we have talked about a few key aspects of web scraping to create a more robust scraper. These include:
- rate limiting
- error handling
- crawl delays
- user agent headers
- robots.txt checking

We can now combine all these aspects into a single scraper class!

We will start by also adding logging to track what’s happening. First, import the logging module:

This creates a logger object that we can use to cleanly log information regarding our scraper’s activity instead of using print statements and feeding to stdout. We set level = logging.INFO to set the level of information we want in our output. There are six levels in order of increasing severity:
- NOTSET=0
- DEBUG=10
- INFO=20
- WARN=30
- ERROR=40
- CRITICAL=50

By setting it to INFO, we will only see messages at the INFO level and above (we wont see any DEBUG messages, for example). Sometimes you may want a more sensitive logger, such as DEBUG, which will log all messages or you may only want to see errors, in which case you would set it to ERROR.

We can then specify the format of the log messages. Here, we have %(asctime)s - %(levelname)s - %(message)s, which means:
- %(asctime)s: The timestamp of the log message
- %(levelname)s: The level of the log message
- %(message)s: The message of the log message

This is a pretty standard format, but you can customize it to your liking.

We can then specify the name of the logger. Here, we have __name__, which is the name of the module. This is useful if we want to log messages from different modules. If you wanted to make other loggers with different names, you would simply change __name__ to the name of the logger you want (i.e. 'api_logger') and can load that logger in different parts of your code.

You can read more about logging here on the official documentation or here with some good explanations for logging.

Lets get started on making our scraper class! We will start by defining the class and its constructor:

Here we are going to take in the following parameters:
- base_url: The base URL of the website to scrape.
- user_agent: The user agent to use for the scraper. If not provided, we will use a default user agent.
- default_delay: The default delay between requests.

We also initialize the session object and the robots parser which will be used to check the robots.txt file.

We then set the headers for the session allowing the user to pass in an agent or use a default user agent accepting HTML, XHTML, and XML content.

Now that we have the headers set and the session object initialized, we are just about ready to start scraping. But, we also need to check the robots.txt file like we did last question to make sure we are not violating any rules set by the website before scraping.

We will do this by creating a private method called _check_robots_txt.

The logic we will use here is similar to what we did last question. We will get the robots.txt file, parse it, and save it to our class’s robots_parser object if one exists. If not, we will log a warning and assume no rules are set.

Now we can add this method to our constructor:

Note that this does not check for permissions yet, we will add that when we start scraping.

Now we will add the core functionality of our scraper. We will start by adding the fetch method which will be used to fetch a URL using the aspects we covered earlier like exponential backoff and error handling (this method will only be used to fetch the content of the page, not to parse the data; that will be done in another method we will create later).

First things first, we need to check if the URL is allowed to be scraped according to the robots.txt file. We will do this by creating a private method called _can_fetch which will return True if the URL is allowed to be scraped, otherwise False.

Lets make a quick private method called _can_fetch to check if the URL is allowed to be scraped:

Nice! Now if the URL is allowed to be scraped, we can proceed to fetch the URL.

Recall that even though we are allowed to scrape the URL, we still need to respect the crawl delay specified in the robots.txt file. We will do this by creating a private method called _get_crawl_delay which will return the crawl delay if it exists, otherwise the default delay.

Now all thats left for us to do here is to handle the rate limiting and server errors! Please use the logic from the previous question to handle the rate limiting and server errors but this time use our logger object to log the messages (log.info for success, log.warning for warnings, and log.error for errors).

We now have functionality for fetching the URL with respect to the websites rules and handle errors gracefully, now lets create a method to parse the data we just fetched.

This will be the main driver method called scrape_quotes that will be used to control the flow of our scraper and will be the entry point for the user to scrape the data they want.

Lets create the scrape_quotes method which will parse the data we just fetched:

Now we can finally parse the data we just fetched.

Our class should be finished now, lets test it out!

Now, create your own robust scraper class that:
1. Checks robots.txt before scraping,
2. Implements exponential backoff,
3. Handles errors gracefully,
4. Includes logging,
5. Respects crawl delays,
6. Uses custom headers.

Use it to scrape multiple pages from a website of your choice.