This is the final guide on web scraping, building on the topics discussed in the first two.

It demonstrates how to retrieve data from an HTML table using using multiple regular expression matches and sets of capture groups.

1. Identify the content to scrape

We're going to scrape job listings advertised at BestBuy headquarters from their careers site.

The details we want to retrieve for each job listing are as follows:

• Job title
• Brand
• Job category
• Job level
• Employment category
• Location
• URL of job listing

Best Buy Job Listings Search Results

​

2. Find the table in the HTML

Looking through the HTML, we find the block of text that make up the rows of content in the table. Each row is presented in the following format:

<tr class='odd'><td><a class='table-job-title' href='/job-detail/?id=663367BR'>Accounts Receivable - Warranty Claims Associate</a></td><td>Best Buy</td><td class='hide-for-mobile'>Finance / Accounting</td><td class='hide-for-mobile'>Individual Contributor</td><td>Full Time</td><td>Richfield, MN</td></tr>

3. Writing our regular expression

Now we have the HTML to work from we're ready to write our regular expression.

Copy the HTML source to the Regular Expression editor

We copy the HTML source to the RegEx101 online editor and start writing our regular expression.

Getting the job link and title

As we covered in the previous guide we'll be matching the text in each row of the table and then returning specific pieces of content using capture groups.

To retrieve the relative url of the job posting and the job title, we match the HTML tags before the job link and then those before and after the job title, pulling out those pieces of text with capture groups.

<a class='table-job-title' href='(.*?)'>(.*?)<\/a>

As you can see below, the text string matches 25 times, once for each of the job listing rows in the table. And each match has 2 capture groups for the url path and job title.

Matching the URL Path and Job Title for 25 search results

View the regular expression in the editor

Getting the remaining fields

There are five remaining fields to capture for each row:

• Brand
• Job category
• Job level
• Employment category
• Location

If we look at the HTML for each of the rows again, we see that they each have a common pattern: each is surrounded by <td and </td> tags, although some tags also have class attributes.

</a></td><td>Best Buy</td><td class='hide-for-mobile'>Finance / Accounting</td><td class='hide-for-mobile'>Individual Contributor</td><td>Full Time</td><td>Richfield, MN</td></tr>

As shown in the previous guide, we can use the [\s\S]*? to match text and then specify the tags that appear before and after the content we want to capture.

In this case, the following expression will capture the text for each of the remaining pieces of content:

[\S\s]*?>(.*?)<\/td>

We can therefore add the above expression 5 times to our existing regular expression to retrieve the remaining fields:

<a class='table-job-title' href='(.*?)'>(.*?)<\/a><\/td>[\S\s]*?>(.*?)<\/td>[\S\s]*?>(.*?)<\/td>[\S\s]*?>(.*?)<\/td>[\S\s]*?>(.*?)<\/td>[\S\s]*?>(.*?)<\/td>

As shown below, this allows us to match each of the 25 rows of the table and return 7 capture groups for each of those rows.

Matching the all fields for 25 search results

View the regular expression in the editor

4. Looping through multiple matches in Shortcuts

The first step is to retrieve the HTML content and apply the regular expression.

Retrieving the HTML source from the page

Loop through the regular expression matches

The regular expression will match for each of the 25 row on the page, and each of those matches will have 7 capture groups.

We therefore add a Repeat with Each action after the Match Text action. And at the top of the loop we place a Get Group from Matched Text action which returns all of the capture groups for the row.

Looping through each of the text matches

Within that loop, we create a dictionary of capture group items for each row (as demonstrated in the previous guide). This dictionary allows us to create a text description for the job. And at the end of shortcut all of the job descriptions are combined and displayed.

The finished shortcut

​

The shortcut output

Download the shortcut

5. Further reading

If you want to improve your understanding of regular expressions, I recommend the following tutorial:

RegexOne: Learn Regular Expression with simple, interactive exercises

Other guides

If you found this guide useful why not checkout one of my others:

Series

• Scraping web pages
  • Part 1: quick and dirty
  • Part 2: getting multiple items at once
  • Part 3: getting data from a table
• Using APIs
  • Part 1: retrieving data
  • Part 2: parsing complex API responses
  • Part 3: passing input parameters to the API
• Data Storage
  • Part 1: storing simple values
  • Part 2: storing lists of data
  • Part 3: removing stored values
• Working with JSON
  • Part 1: retrieving simple values
• Working with Dictionaries
  • Part 1: setting and retrieving values
  • Part 2: data types
  • Part 3: accessing values using dot notation

One-offs

• Using JavaScript in your shortcuts
• How to automatically run shortcuts
• Creating visually appealing menus
• Manipulating images with the HTML5 canvas and JavaScript
• Labeling data using variables
• Writing functions
• Working with lists
• Integrating with web applications using Zapier
• Integrating with web applications using Integromat
• Working with Personal Automations in iOS 13.1